[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:33 dist-test-slave-6gxc krb5kdc[10294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:33 dist-test-slave-6gxc krb5kdc[10294](info): set up 2 sockets
Sep 27 00:33:33 dist-test-slave-6gxc krb5kdc[10294](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:33:34.491474 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:34 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:34.606806 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34145
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:34.710767 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:34.710978 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:34.711043 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:34.714972 10307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:34.715018 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:34.715036 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:34.715070 10307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:34.715099 10307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:34.720058 10307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:34.720907 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:34.721778 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:34.729295 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:34.729303 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:34.729409 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:34.729473 10307 server_base.cc:1034] running on GCE node
I20260927 00:33:34.730204 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:34.731240 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:34.732415 10307 hybrid_clock.cc:648] HybridClock initialized: now 1790469214732403 us; error 26 us; skew 500 ppm
Sep 27 00:33:34 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469214, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:34.735071 10307 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:34.735841 10307 webserver.cc:460] Webserver started at http://127.10.11.254:39819/ using document root <none> and password file <none>
I20260927 00:33:34.736348 10307 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:34.736429 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:34.736632 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:34.738552 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "8b1d8c10cc624d9e9cc9021699aea74d"
format_stamp: "Formatted at 2026-09-27 00:33:34 on dist-test-slave-6gxc"
server_key: "86b8ad64a00520687d97cd17578a6282"
server_key_iv: "3df2d916a15bc14d642a212f8a231c45"
server_key_version: "encryptionkey@0"
I20260927 00:33:34.739180 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "8b1d8c10cc624d9e9cc9021699aea74d"
format_stamp: "Formatted at 2026-09-27 00:33:34 on dist-test-slave-6gxc"
server_key: "86b8ad64a00520687d97cd17578a6282"
server_key_iv: "3df2d916a15bc14d642a212f8a231c45"
server_key_version: "encryptionkey@0"
I20260927 00:33:34.742657 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:33:34.744810 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:34.745939 10307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:34.746073 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "8b1d8c10cc624d9e9cc9021699aea74d"
format_stamp: "Formatted at 2026-09-27 00:33:34 on dist-test-slave-6gxc"
server_key: "86b8ad64a00520687d97cd17578a6282"
server_key_iv: "3df2d916a15bc14d642a212f8a231c45"
server_key_version: "encryptionkey@0"
I20260927 00:33:34.746241 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:34.757247 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:34.760218 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:34.760406 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:34.767030 10307 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:34145
I20260927 00:33:34.767042 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:34145 every 8 connection(s)
I20260927 00:33:34.767895 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:34.770412 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:34.772055 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10307
I20260927 00:33:34.772213 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:34.773603 10287 external_mini_cluster.cc:1428] Setting key ac92874e8a2f0a4257bde73d7da048a8
I20260927 00:33:34.775705 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d: Bootstrap starting.
I20260927 00:33:34.778343 10375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:34.779024 10375 log.cc:824] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:34.780680 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d: No bootstrap required, opened a new log
I20260927 00:33:34.783958 10375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } }
I20260927 00:33:34.784174 10375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:34.784260 10375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1d8c10cc624d9e9cc9021699aea74d, State: Initialized, Role: FOLLOWER
I20260927 00:33:34.784679 10375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } }
I20260927 00:33:34.784824 10375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:34.784902 10375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:34.784993 10375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:34.785791 10375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } }
I20260927 00:33:34.786134 10375 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1d8c10cc624d9e9cc9021699aea74d; no voters:
I20260927 00:33:34.786425 10375 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:34.786514 10380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:34.786723 10380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [term 1 LEADER]: Becoming Leader. State: Replica: 8b1d8c10cc624d9e9cc9021699aea74d, State: Running, Role: LEADER
I20260927 00:33:34.787074 10380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } }
I20260927 00:33:34.787580 10375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:34.788178 10382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1d8c10cc624d9e9cc9021699aea74d. Latest consensus state: current_term: 1 leader_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } } }
I20260927 00:33:34.788298 10382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:34.788911 10381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1d8c10cc624d9e9cc9021699aea74d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34145 } } }
I20260927 00:33:34.788992 10381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:34.790146 10385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 00:33:34 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:34.793007 10385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:34.797873 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:34.776715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.10.11.254:34145)
0927 00:33:34.777144 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:34.777159 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:34.778142 (+ 983us) server_negotiation.cc:407] Connection header received
0927 00:33:34.781222 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:34.781237 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:34.781519 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:34.781829 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:34.783874 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:34.784986 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:34.786252 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:34.786952 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:34.787111 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:34.791856 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:34.791861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:34.791876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:34.791907 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:34.793909 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:34.794546 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:34.794552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:34.794557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:34.794694 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:34.795062 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:34.795066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:34.795068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:34.795544 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:34.795749 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:34.796705 (+ 956us) server_negotiation.cc:299] Negotiation successful
0927 00:33:34.796979 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260927 00:33:34.798050 10385 catalog_manager.cc:1347] Generated new cluster ID: 4082788b579e43b3a19d5a6458d46846
I20260927 00:33:34.798105 10385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:34.824955 10385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:34.825755 10385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:34.847551 10385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1d8c10cc624d9e9cc9021699aea74d: Generated new TSK 0
I20260927 00:33:34.848176 10385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:34.882428 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:34.985635 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:34.985863 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:34.985904 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:34.990088 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:34.990155 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:34.990231 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:34.995148 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:34.996065 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:34.996996 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:35.005990 10407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:35.006089 10409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:35.006258 10406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:35.006361 10401 server_base.cc:1034] running on GCE node
I20260927 00:33:35.006805 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:35.007468 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:35.008675 10401 hybrid_clock.cc:648] HybridClock initialized: now 1790469215008658 us; error 40 us; skew 500 ppm
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:35.011309 10401 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:35.012135 10401 webserver.cc:460] Webserver started at http://127.10.11.193:38595/ using document root <none> and password file <none>
I20260927 00:33:35.012632 10401 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:35.012712 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:35.012920 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:35.014825 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "13dcde81fc084df8b92467f38683880d"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "9cd3a4e4070427f23dadf36b1c5aa214"
server_key_iv: "d8457ce7707822efb797fdfcc17b4606"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.015337 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "13dcde81fc084df8b92467f38683880d"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "9cd3a4e4070427f23dadf36b1c5aa214"
server_key_iv: "d8457ce7707822efb797fdfcc17b4606"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.018635 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:35.020841 10416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.021941 10401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:35.022078 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "13dcde81fc084df8b92467f38683880d"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "9cd3a4e4070427f23dadf36b1c5aa214"
server_key_iv: "d8457ce7707822efb797fdfcc17b4606"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.022229 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:35.053076 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:35.055967 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:35.056147 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:35.056679 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:35.057441 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:35.057507 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.057571 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:35.057616 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.065992 10401 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:38559
I20260927 00:33:35.066008 10529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:38559 every 8 connection(s)
I20260927 00:33:35.066839 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:35.067593 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10401
I20260927 00:33:35.067703 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:35.067950 10287 external_mini_cluster.cc:1428] Setting key b6f98ece2d2e0dd81787d9413670883e
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:33:35.082361 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.068534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:43355 (local address 127.10.11.254:34145)
0927 00:33:35.068666 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:35.068670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:35.069604 (+ 934us) server_negotiation.cc:407] Connection header received
0927 00:33:35.071556 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:35.071562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:35.071623 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:35.071720 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:35.073048 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.073645 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.075568 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.075802 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.075886 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.078645 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:35.078648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:35.078650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:35.078675 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:35.080030 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.080781 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.080785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.080786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.080868 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.081233 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.081236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.081237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.081421 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:35.081525 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:35.082002 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 00:33:35.082182 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:33:35.083154 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.068922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34145 (local address 127.10.11.193:43355)
0927 00:33:35.069440 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:35.069493 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:35.071293 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:35.071864 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:35.071870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:35.072246 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:35.072895 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.072917 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.073775 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.073780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.075431 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.075447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.075958 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.075964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.076187 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.076779 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:35.076801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:35.078491 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:35.080169 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.080175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.080187 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.080608 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.080966 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.080969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.080982 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.081123 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.081523 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:35.081530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:35.081768 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 00:33:35.081956 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 00:33:35.082323 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260927 00:33:35.084506 10530 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34145
I20260927 00:33:35.084790 10530 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:35.085264 10530 heartbeater.cc:511] Master 127.10.11.254:34145 requested a full tablet report, sending...
I20260927 00:33:35.086838 10339 ts_manager.cc:194] Registered new tserver with Master: 13dcde81fc084df8b92467f38683880d (127.10.11.193:38559)
I20260927 00:33:35.088061 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:43355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:35.095080 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:35.204275 10535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:35.204519 10535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:35.204591 10535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:35.208572 10535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:35.208650 10535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:35.208736 10535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:35.213827 10535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:35.214761 10535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:35.215651 10535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:35.224279 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:35.224263 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:35.224421 10535 server_base.cc:1034] running on GCE node
W20260927 00:33:35.224346 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:35.224896 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:35.225404 10535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:35.226550 10535 hybrid_clock.cc:648] HybridClock initialized: now 1790469215226533 us; error 49 us; skew 500 ppm
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:35.229079 10535 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:35.229839 10535 webserver.cc:460] Webserver started at http://127.10.11.194:36911/ using document root <none> and password file <none>
I20260927 00:33:35.230378 10535 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:35.230458 10535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:35.230659 10535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:35.232481 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "80cc2fa81e30490dbde84c7e8559c352"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "92d7b9351869c982f0d26b869d3bf57c"
server_key_iv: "18a486f3c88338c4a5c7ae995c31e49a"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.233013 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "80cc2fa81e30490dbde84c7e8559c352"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "92d7b9351869c982f0d26b869d3bf57c"
server_key_iv: "18a486f3c88338c4a5c7ae995c31e49a"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.236306 10535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:35.238435 10550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.239504 10535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:35.239624 10535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "80cc2fa81e30490dbde84c7e8559c352"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "92d7b9351869c982f0d26b869d3bf57c"
server_key_iv: "18a486f3c88338c4a5c7ae995c31e49a"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.239719 10535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:35.262773 10535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:35.265461 10535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:35.265635 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:35.266187 10535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:35.266904 10535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:35.266968 10535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.267036 10535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:35.267082 10535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.275554 10535 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:41797
I20260927 00:33:35.275578 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:41797 every 8 connection(s)
I20260927 00:33:35.276295 10535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:35.280345 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10535
I20260927 00:33:35.280481 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:35.280717 10287 external_mini_cluster.cc:1428] Setting key b8fd931f3243e3a8daf841acb711df56
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:33:35.290405 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.278000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:46015 (local address 127.10.11.254:34145)
0927 00:33:35.278231 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:35.278234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:35.278946 (+ 712us) server_negotiation.cc:407] Connection header received
0927 00:33:35.280161 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:35.280164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:35.280216 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:35.280317 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:35.281559 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.282199 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.283298 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.283571 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.283656 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.286312 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:35.286326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:35.286332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:35.286363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:35.288052 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.288724 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.288730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.288735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.288830 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.289249 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.289253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.289254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.289522 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:35.289627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:35.290048 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 00:33:35.290187 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:33:35.291082 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.278390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34145 (local address 127.10.11.194:46015)
0927 00:33:35.278807 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:35.278844 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:35.279974 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:35.280492 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:35.280499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:35.280912 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:35.281433 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.281448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.282319 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.282322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.283161 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.283170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.283711 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.283716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.283955 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.284509 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:35.284524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:35.286159 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:35.288245 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.288271 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.288281 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.288621 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.288931 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.288967 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.288969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.289145 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.289622 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:35.289627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:35.289829 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 00:33:35.289977 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 00:33:35.290303 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260927 00:33:35.292143 10664 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34145
I20260927 00:33:35.292354 10664 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:33:35.292815 10664 heartbeater.cc:511] Master 127.10.11.254:34145 requested a full tablet report, sending...
I20260927 00:33:35.293852 10339 ts_manager.cc:194] Registered new tserver with Master: 80cc2fa81e30490dbde84c7e8559c352 (127.10.11.194:41797)
I20260927 00:33:35.294674 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:46015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:35.309523 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:35.413580 10669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:35.413861 10669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:35.413952 10669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:35.417838 10669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:35.417915 10669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:35.417999 10669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:35.422827 10669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:35.423683 10669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:35.424544 10669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:35.433650 10677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:35.433619 10674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:35.433696 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:35.433876 10669 server_base.cc:1034] running on GCE node
I20260927 00:33:35.434360 10669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:35.434918 10669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:35.436091 10669 hybrid_clock.cc:648] HybridClock initialized: now 1790469215436064 us; error 46 us; skew 500 ppm
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:35.438700 10669 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:35.439670 10669 webserver.cc:460] Webserver started at http://127.10.11.195:35905/ using document root <none> and password file <none>
I20260927 00:33:35.440323 10669 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:35.440383 10669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:35.440531 10669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:35.442351 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "b16947631b4045c1824b93c8a3a11b45"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "56f0b1fcb31de408a9442e722c4efdc9"
server_key_iv: "1eb4a6d2e64791d2ecd56ce52e145ebd"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.442868 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "b16947631b4045c1824b93c8a3a11b45"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "56f0b1fcb31de408a9442e722c4efdc9"
server_key_iv: "1eb4a6d2e64791d2ecd56ce52e145ebd"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.446354 10669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:35.448436 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.449574 10669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:35.449681 10669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "b16947631b4045c1824b93c8a3a11b45"
format_stamp: "Formatted at 2026-09-27 00:33:35 on dist-test-slave-6gxc"
server_key: "56f0b1fcb31de408a9442e722c4efdc9"
server_key_iv: "1eb4a6d2e64791d2ecd56ce52e145ebd"
server_key_version: "encryptionkey@0"
I20260927 00:33:35.449780 10669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:35.470296 10669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:35.472916 10669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:35.473049 10669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:35.473493 10669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:35.474225 10669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:35.474274 10669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.474310 10669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:35.474330 10669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:35.482328 10669 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:42443
I20260927 00:33:35.482388 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:42443 every 8 connection(s)
I20260927 00:33:35.483098 10669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:35.484432 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10669
I20260927 00:33:35.484556 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:35.484840 10287 external_mini_cluster.cc:1428] Setting key 7cda9bd69937ce22836e04580664d7e3
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:35.495853 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.485461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:51377 (local address 127.10.11.254:34145)
0927 00:33:35.485578 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:35.485582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:35.485761 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:33:35.487087 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:35.487090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:35.487144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:35.487240 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:35.488419 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.489054 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.490239 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.490465 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.490568 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.492672 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:35.492676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:35.492679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:35.492706 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:35.493917 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.494433 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.494437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.494439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.494541 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.494865 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.494868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.494869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.495042 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:35.495139 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:35.495583 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:33:35.495690 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:33:35.496498 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.485155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34145 (local address 127.10.11.195:51377)
0927 00:33:35.485624 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:35.485670 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:35.486893 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:35.487428 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:35.487435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:35.487832 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:35.488290 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.488303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.489165 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.489168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.490079 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:35.490088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.490596 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.490600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:35.490766 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.491208 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:35.491223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:35.492529 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:35.494026 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.494031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.494043 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.494338 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.494632 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:35.494636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:35.494638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:35.494769 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:35.495150 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:35.495157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:35.495365 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:33:35.495527 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 00:33:35.495812 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260927 00:33:35.497398 10798 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34145
I20260927 00:33:35.497634 10798 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:35.498071 10798 heartbeater.cc:511] Master 127.10.11.254:34145 requested a full tablet report, sending...
I20260927 00:33:35.499027 10339 ts_manager.cc:194] Registered new tserver with Master: b16947631b4045c1824b93c8a3a11b45 (127.10.11.195:42443)
I20260927 00:33:35.499550 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:51377
I20260927 00:33:35.510327 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:35.535250 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.525939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.10.11.193:38559)
0927 00:33:35.526288 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:35.526304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:35.526330 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:33:35.526530 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:35.526535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:35.526713 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:35.526856 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:35.527573 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.528343 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.529106 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.529446 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.529527 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.531809 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:35.531816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:35.531827 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:35.531847 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:35.533286 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.533724 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.533729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.533730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.533844 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.534229 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.534232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.534234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.534535 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:35.534697 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:35.534904 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 00:33:35.535078 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
W20260927 00:33:35.536182 10444 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:35.561077 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:35.552724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.10.11.193:38559)
0927 00:33:35.552933 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:35.552937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:35.552992 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:33:35.553097 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:35.553099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:35.553138 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:35.553222 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:35.553897 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.554508 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.555315 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:35.555535 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:35.555614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:35.558093 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:35.558176 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:35.558178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:35.558202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:35.559590 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.559975 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.559978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.559979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.560054 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:35.560295 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:35.560298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:35.560299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:35.560463 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:35.560569 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:35.560756 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:33:35.560851 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 00:33:35.562144 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10401
I20260927 00:33:35.567399 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10535
I20260927 00:33:35.572728 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10669
I20260927 00:33:35.577723 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10307
2026-09-27T00:33:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2139 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10812](info): set up 2 sockets
Sep 27 00:33:35 dist-test-slave-6gxc krb5kdc[10812](info): commencing operation
krb5kdc: starting...
W20260927 00:33:36.614025 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:36 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:36.728420 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34033
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:36.833891 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:36.834213 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:36.834256 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:36.838136 10825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:36.838182 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:36.838201 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:36.838218 10825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:36.838233 10825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:36.843334 10825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:36.844245 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:36.845145 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:36.853039 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:36.853027 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:36.853026 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:36.853395 10825 server_base.cc:1034] running on GCE node
I20260927 00:33:36.853885 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:36.854933 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:36.856132 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790469216856106 us; error 36 us; skew 500 ppm
Sep 27 00:33:36 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469216, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:36.861934 10825 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:36.862823 10825 webserver.cc:460] Webserver started at http://127.10.11.254:41981/ using document root <none> and password file <none>
I20260927 00:33:36.863399 10825 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:36.863482 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:36.863699 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:36.865640 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "ce0f7905373f418e891a481448fa176d"
format_stamp: "Formatted at 2026-09-27 00:33:36 on dist-test-slave-6gxc"
server_key: "f89a02e13e933d2e050daf5fb45a71d5"
server_key_iv: "9de7bb17c6efcd988a0cddc8b276cc2e"
server_key_version: "encryptionkey@0"
I20260927 00:33:36.866243 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "ce0f7905373f418e891a481448fa176d"
format_stamp: "Formatted at 2026-09-27 00:33:36 on dist-test-slave-6gxc"
server_key: "f89a02e13e933d2e050daf5fb45a71d5"
server_key_iv: "9de7bb17c6efcd988a0cddc8b276cc2e"
server_key_version: "encryptionkey@0"
I20260927 00:33:36.869664 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:33:36.871886 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:36.872956 10825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:36.873082 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "ce0f7905373f418e891a481448fa176d"
format_stamp: "Formatted at 2026-09-27 00:33:36 on dist-test-slave-6gxc"
server_key: "f89a02e13e933d2e050daf5fb45a71d5"
server_key_iv: "9de7bb17c6efcd988a0cddc8b276cc2e"
server_key_version: "encryptionkey@0"
I20260927 00:33:36.873196 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:36.896719 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:36.899770 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:36.899957 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:36.906450 10825 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:34033
I20260927 00:33:36.906466 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:34033 every 8 connection(s)
I20260927 00:33:36.907364 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:36.909945 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:36.913832 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10825
I20260927 00:33:36.913944 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:36.914238 10287 external_mini_cluster.cc:1428] Setting key d2b028cb14b917042f2785759e705bff
I20260927 00:33:36.915524 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d: Bootstrap starting.
I20260927 00:33:36.917626 10893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:36.918375 10893 log.cc:824] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:36.920090 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d: No bootstrap required, opened a new log
Sep 27 00:33:36 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:36.923750 10893 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } }
I20260927 00:33:36.924017 10893 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:36.924103 10893 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0f7905373f418e891a481448fa176d, State: Initialized, Role: FOLLOWER
I20260927 00:33:36.924564 10893 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } }
I20260927 00:33:36.924698 10893 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:36.924793 10893 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:36.924889 10893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:36.925655 10893 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } }
I20260927 00:33:36.925983 10893 leader_election.cc:302] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0f7905373f418e891a481448fa176d; no voters:
I20260927 00:33:36.926294 10893 leader_election.cc:288] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:36.926388 10898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:36.926601 10898 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [term 1 LEADER]: Becoming Leader. State: Replica: ce0f7905373f418e891a481448fa176d, State: Running, Role: LEADER
I20260927 00:33:36.926877 10898 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } }
I20260927 00:33:36.927542 10893 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:36.928467 10900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0f7905373f418e891a481448fa176d. Latest consensus state: current_term: 1 leader_uuid: "ce0f7905373f418e891a481448fa176d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } } }
I20260927 00:33:36.928587 10900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:36.928861 10899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0f7905373f418e891a481448fa176d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0f7905373f418e891a481448fa176d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34033 } } }
I20260927 00:33:36.928905 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:36.915653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.10.11.254:34033)
0927 00:33:36.916043 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:36.916060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:36.916095 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:33:36.916691 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:36.916706 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:36.917022 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:36.917345 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:36.918124 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:36.919304 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:36.920120 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:36.920625 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:36.920800 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:36.923446 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:36.923450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:36.923463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:36.923487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:36.925343 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:36.925757 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:36.925762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:36.925768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:36.925900 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:36.926195 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:36.926200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:36.926205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:36.926722 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:36.927475 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:36.927624 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 00:33:36.927909 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260927 00:33:36.928947 10899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:36.929186 10907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:36.932169 10907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:36.937346 10907 catalog_manager.cc:1347] Generated new cluster ID: 175e597c90354abd8b2f630530df1f6e
I20260927 00:33:36.937402 10907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:36.950718 10907 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:36.951884 10907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:36.964931 10907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0f7905373f418e891a481448fa176d: Generated new TSK 0
I20260927 00:33:36.965726 10907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:36.996443 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:37.104112 10919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:37.104359 10919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:37.104431 10919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:37.108335 10919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:37.108409 10919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:37.108496 10919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:37.113480 10919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:37.114391 10919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:37.115316 10919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:37.124150 10924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:37.124158 10927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:37.124158 10925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:37.124383 10919 server_base.cc:1034] running on GCE node
I20260927 00:33:37.124799 10919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:37.125349 10919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:37.126526 10919 hybrid_clock.cc:648] HybridClock initialized: now 1790469217126510 us; error 27 us; skew 500 ppm
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:37.129498 10919 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:37.130411 10919 webserver.cc:460] Webserver started at http://127.10.11.193:40739/ using document root <none> and password file <none>
I20260927 00:33:37.130960 10919 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:37.131026 10919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:37.131289 10919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:37.133307 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "b658186909084157946c3f2d70fbe56e"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "e4e495d1ba7dcec64f12eac3978de964"
server_key_iv: "ca606266361e8f3dd85c7ce93fbb4b37"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.134081 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "b658186909084157946c3f2d70fbe56e"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "e4e495d1ba7dcec64f12eac3978de964"
server_key_iv: "ca606266361e8f3dd85c7ce93fbb4b37"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.137907 10919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:37.140586 10934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.141919 10919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:37.142062 10919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "b658186909084157946c3f2d70fbe56e"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "e4e495d1ba7dcec64f12eac3978de964"
server_key_iv: "ca606266361e8f3dd85c7ce93fbb4b37"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.142230 10919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:37.158025 10919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:37.161140 10919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:37.161450 10919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:37.162019 10919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:37.162868 10919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:37.162914 10919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.162978 10919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:37.163007 10919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.171927 10919 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:39707
I20260927 00:33:37.172645 11047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:39707 every 8 connection(s)
I20260927 00:33:37.172912 10919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:37.183128 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 10919
I20260927 00:33:37.183239 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:37.183508 10287 external_mini_cluster.cc:1428] Setting key cecebffb9057e4ec6538c0e9bda7c34e
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:37.189875 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.177471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44599 (local address 127.10.11.254:34033)
0927 00:33:37.177633 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.177637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.178977 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 00:33:37.180091 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.180094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.180168 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.180256 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.181604 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.182237 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.183571 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.183808 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.183920 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.186511 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.186514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.186517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.186541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.187943 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.188457 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.188461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.188463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.188539 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.188856 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.188860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.188862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.189028 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.189130 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.189574 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.189700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:33:37.190649 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.177958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34033 (local address 127.10.11.193:44599)
0927 00:33:37.178828 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.178868 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.179856 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.180554 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.180564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.180903 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:37.181440 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.181462 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.182348 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.182350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.183450 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.183459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.183956 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.183959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.184143 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.184731 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:37.184750 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:37.186364 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:37.188064 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.188069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.188082 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.188370 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.188626 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.188630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.188632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.188762 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.189126 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:37.189131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:37.189360 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.189534 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.189822 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":152,"threads_started":1}
I20260927 00:33:37.191763 11048 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34033
I20260927 00:33:37.191987 11048 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:33:37.192441 11048 heartbeater.cc:511] Master 127.10.11.254:34033 requested a full tablet report, sending...
I20260927 00:33:37.193768 10857 ts_manager.cc:194] Registered new tserver with Master: b658186909084157946c3f2d70fbe56e (127.10.11.193:39707)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:33:37.195040 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:37.208525 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:37.311609 11053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:37.311822 11053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:37.311863 11053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:37.315755 11053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:37.315801 11053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:37.315894 11053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:37.321019 11053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:37.321878 11053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:37.322837 11053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:37.331877 11058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:37.331959 11061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:37.331931 11053 server_base.cc:1034] running on GCE node
W20260927 00:33:37.332070 11059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:37.332417 11053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:37.332966 11053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:37.334263 11053 hybrid_clock.cc:648] HybridClock initialized: now 1790469217334248 us; error 27 us; skew 500 ppm
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:37.336881 11053 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:37.337663 11053 webserver.cc:460] Webserver started at http://127.10.11.194:39589/ using document root <none> and password file <none>
I20260927 00:33:37.338228 11053 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:37.338312 11053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:37.338510 11053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:37.340328 11053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "1eaa35ab205eef83e4c6c4caa2ad977d"
server_key_iv: "9c4f2f3f7e6d6fe79d8ac7179bac9f6d"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.340860 11053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "1eaa35ab205eef83e4c6c4caa2ad977d"
server_key_iv: "9c4f2f3f7e6d6fe79d8ac7179bac9f6d"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.344184 11053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:37.346508 11068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.347649 11053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:33:37.347805 11053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "1eaa35ab205eef83e4c6c4caa2ad977d"
server_key_iv: "9c4f2f3f7e6d6fe79d8ac7179bac9f6d"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.347909 11053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:37.372813 11053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:37.375684 11053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:37.375885 11053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:37.376456 11053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:37.377251 11053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:37.377321 11053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.377388 11053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:37.377427 11053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.385669 11053 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:46187
I20260927 00:33:37.385728 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:46187 every 8 connection(s)
I20260927 00:33:37.386616 11053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:37.393628 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11053
I20260927 00:33:37.393754 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:37.394068 10287 external_mini_cluster.cc:1428] Setting key 34801f810a74c5a9ceeceee08887bd57
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:37.400440 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.388301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:47095 (local address 127.10.11.254:34033)
0927 00:33:37.388421 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.388425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.389401 (+ 976us) server_negotiation.cc:407] Connection header received
0927 00:33:37.390288 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.390292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.390347 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.390451 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.391817 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.392362 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.393556 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.393863 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.393957 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.396589 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.396593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.396600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.396623 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.398246 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.398727 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.398733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.398738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.398844 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.399189 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.399195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.399199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.399453 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.399564 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.400082 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.400216 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:33:37.401029 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.388613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34033 (local address 127.10.11.194:47095)
0927 00:33:37.389243 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.389280 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.390056 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.390606 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.390613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.391179 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:37.391649 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.391662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.392504 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.392508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.393417 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.393429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.394015 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.394020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.394243 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.394837 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:37.394858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:37.396430 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:37.398364 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.398368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.398378 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.398611 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.398958 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.398961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.398962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.399088 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.399580 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:37.399585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:37.399789 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.399999 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.400225 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":116,"threads_started":1}
I20260927 00:33:37.402145 11182 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34033
I20260927 00:33:37.402364 11182 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:37.402787 11182 heartbeater.cc:511] Master 127.10.11.254:34033 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:33:37.403769 10857 ts_manager.cc:194] Registered new tserver with Master: 22b3e5b547b4429eaf3ec4d42b0e4a38 (127.10.11.194:46187)
I20260927 00:33:37.404296 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:47095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:37.420696 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:37.525478 11187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:37.525694 11187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:37.525734 11187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:37.529793 11187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:37.529839 11187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:37.529932 11187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:37.534893 11187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34033
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:37.535773 11187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:37.536659 11187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:37.545506 11195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:37.545498 11192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:37.545511 11193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:37.545922 11187 server_base.cc:1034] running on GCE node
I20260927 00:33:37.546307 11187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:37.546952 11187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:37.548101 11187 hybrid_clock.cc:648] HybridClock initialized: now 1790469217548090 us; error 24 us; skew 500 ppm
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:37.550683 11187 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:37.551435 11187 webserver.cc:460] Webserver started at http://127.10.11.195:33913/ using document root <none> and password file <none>
I20260927 00:33:37.551962 11187 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:37.552021 11187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:37.552168 11187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:37.553949 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "c74de56ef61f4571a4c60974a8170842"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "d301df5e92947844dae320eee5ad9921"
server_key_iv: "ea674659a730b90073fbcff26636c5c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.554481 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "c74de56ef61f4571a4c60974a8170842"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "d301df5e92947844dae320eee5ad9921"
server_key_iv: "ea674659a730b90073fbcff26636c5c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.557725 11187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:37.559943 11202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.561035 11187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:37.561123 11187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "c74de56ef61f4571a4c60974a8170842"
format_stamp: "Formatted at 2026-09-27 00:33:37 on dist-test-slave-6gxc"
server_key: "d301df5e92947844dae320eee5ad9921"
server_key_iv: "ea674659a730b90073fbcff26636c5c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:37.561194 11187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:37.582441 11187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:37.584877 11187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:37.585007 11187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:37.585448 11187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:37.586154 11187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:37.586197 11187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.586233 11187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:37.586253 11187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:37.594698 11187 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:34231
I20260927 00:33:37.594761 11315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:34231 every 8 connection(s)
I20260927 00:33:37.595507 11187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:37.595851 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11187
I20260927 00:33:37.595969 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:37.596241 10287 external_mini_cluster.cc:1428] Setting key f92bf574b8be526ef0c90ac4cf87b30b
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:37.607359 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.597158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:43217 (local address 127.10.11.254:34033)
0927 00:33:37.597286 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.597291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.598171 (+ 880us) server_negotiation.cc:407] Connection header received
0927 00:33:37.599064 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.599066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.599108 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.599175 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.600231 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.600729 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.601762 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.601987 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.602073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.604280 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.604282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.604284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.604306 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.605689 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.606154 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.606156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.606158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.606242 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.606515 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.606517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.606518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.606669 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.606773 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.607162 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.607250 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:33:37.608036 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.597560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34033 (local address 127.10.11.195:43217)
0927 00:33:37.598021 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.598058 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.598894 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.599333 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.599339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.599697 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:37.600118 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.600130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.600829 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.600832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.601647 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.601655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.602144 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.602149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.602292 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.602765 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:37.602780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:37.604147 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:37.605800 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.605804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.605814 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.606034 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.606320 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:37.606322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:37.606323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:37.606440 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:37.606784 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:37.606788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:37.606984 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.607122 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.607334 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260927 00:33:37.609231 11316 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34033
I20260927 00:33:37.609463 11316 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:37.609980 11316 heartbeater.cc:511] Master 127.10.11.254:34033 requested a full tablet report, sending...
I20260927 00:33:37.610924 10857 ts_manager.cc:194] Registered new tserver with Master: c74de56ef61f4571a4c60974a8170842 (127.10.11.195:34231)
I20260927 00:33:37.611445 10857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:43217
I20260927 00:33:37.621392 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:33:37.631412 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.624473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.10.11.254:34033)
0927 00:33:37.624577 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.624580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.624695 (+ 115us) server_negotiation.cc:407] Connection header received
0927 00:33:37.624792 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.624794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.624827 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.624891 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.625462 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.625943 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.626736 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.626971 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.627077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.628337 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.628340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.628342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.628361 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.629800 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.630203 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.630207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.630209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.630287 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.630544 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.630549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.630550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.630743 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.630905 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.631158 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.631265 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:33:37.635370 10857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:37.638223 10857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:37.651741 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.645729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:46187 (local address 127.0.0.1:44728)
0927 00:33:37.647096 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.647108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.647199 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.647752 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.647754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.647770 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:37.647877 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.647883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.649207 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.649210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.650426 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.650433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651219 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.651349 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.651388 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.651492 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.651591 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":21,"thread_start_us":93,"threads_started":1}
I20260927 00:33:37.651873 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.645595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:34231 (local address 127.0.0.1:35186)
0927 00:33:37.646479 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.646525 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.646661 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.647596 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.647602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.647622 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:37.647739 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.647746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.649151 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.649154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.650233 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.650242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651387 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.651515 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.651539 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.651596 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.651664 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":76,"threads_started":1}
I20260927 00:33:37.652277 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.646331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.10.11.195:34231)
0927 00:33:37.647092 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.647104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.647128 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:33:37.647200 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.647206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.647396 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.647597 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:37.647869 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.649025 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.650351 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651226 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651855 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.651995 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.652044 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.652139 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":120,"threads_started":1}
I20260927 00:33:37.652287 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.645919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.10.11.194:46187)
0927 00:33:37.646542 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.646555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.647180 (+ 625us) server_negotiation.cc:407] Connection header received
0927 00:33:37.647307 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.647311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.647477 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.647600 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:37.648001 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.649089 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.650508 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651094 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651222 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.651429 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.652106 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.652187 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":171,"threads_started":1}
I20260927 00:33:37.652940 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.646090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39707 (local address 127.0.0.1:46928)
0927 00:33:37.647338 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.647348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.647443 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.647889 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.647892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.647906 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:37.647989 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.647997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.648921 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.648925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.650311 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.650329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651727 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.652433 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.652675 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.652740 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.652814 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":53,"thread_start_us":78,"threads_started":1}
I20260927 00:33:37.653319 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.646684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.10.11.193:39707)
0927 00:33:37.646934 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.646966 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.647429 (+ 463us) server_negotiation.cc:407] Connection header received
0927 00:33:37.647550 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.647556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.647742 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.647880 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:37.648077 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.648789 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.650635 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.651442 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.651533 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.651670 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.653104 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.653190 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260927 00:33:37.654956 11116 tablet_service.cc:1467] Processing CreateTablet for tablet 359514f904864024b284a6b4b434222d (DEFAULT_TABLE table=test-table [id=88e7b258073a4575aa56b7f1c00d85e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:37.655030 11250 tablet_service.cc:1467] Processing CreateTablet for tablet 359514f904864024b284a6b4b434222d (DEFAULT_TABLE table=test-table [id=88e7b258073a4575aa56b7f1c00d85e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:37.655776 11116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359514f904864024b284a6b4b434222d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:37.655975 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359514f904864024b284a6b4b434222d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:37.655833 10982 tablet_service.cc:1467] Processing CreateTablet for tablet 359514f904864024b284a6b4b434222d (DEFAULT_TABLE table=test-table [id=88e7b258073a4575aa56b7f1c00d85e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:37.656636 10982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359514f904864024b284a6b4b434222d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:37.661352 11333 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Bootstrap starting.
I20260927 00:33:37.663072 11335 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Bootstrap starting.
I20260927 00:33:37.663385 11334 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Bootstrap starting.
I20260927 00:33:37.664330 11333 tablet_bootstrap.cc:654] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:37.665097 11335 tablet_bootstrap.cc:654] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:37.665228 11334 tablet_bootstrap.cc:654] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:37.665601 11333 log.cc:824] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:37.666267 11335 log.cc:824] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:37.666338 11334 log.cc:824] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:37.668040 11335 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: No bootstrap required, opened a new log
I20260927 00:33:37.668051 11333 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: No bootstrap required, opened a new log
I20260927 00:33:37.668068 11334 tablet_bootstrap.cc:492] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: No bootstrap required, opened a new log
I20260927 00:33:37.668226 11333 ts_tablet_manager.cc:1397] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:33:37.668226 11334 ts_tablet_manager.cc:1397] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:33:37.668229 11335 ts_tablet_manager.cc:1397] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:33:37.670886 11335 raft_consensus.cc:348] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.671129 11335 raft_consensus.cc:374] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:37.671186 11335 raft_consensus.cc:729] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b658186909084157946c3f2d70fbe56e, State: Initialized, Role: FOLLOWER
I20260927 00:33:37.671607 11335 consensus_queue.cc:260] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.671878 11333 raft_consensus.cc:348] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.671878 11334 raft_consensus.cc:348] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.672122 11333 raft_consensus.cc:374] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:37.672122 11334 raft_consensus.cc:374] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:37.672166 11048 heartbeater.cc:503] Master 127.10.11.254:34033 was elected leader, sending a full tablet report...
I20260927 00:33:37.672170 11334 raft_consensus.cc:729] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74de56ef61f4571a4c60974a8170842, State: Initialized, Role: FOLLOWER
I20260927 00:33:37.672178 11333 raft_consensus.cc:729] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b3e5b547b4429eaf3ec4d42b0e4a38, State: Initialized, Role: FOLLOWER
I20260927 00:33:37.672503 11335 ts_tablet_manager.cc:1428] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260927 00:33:37.672600 11333 consensus_queue.cc:260] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.672600 11334 consensus_queue.cc:260] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.673172 11316 heartbeater.cc:503] Master 127.10.11.254:34033 was elected leader, sending a full tablet report...
I20260927 00:33:37.673460 11334 ts_tablet_manager.cc:1428] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:33:37.674331 11333 ts_tablet_manager.cc:1428] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:33:37.674608 11182 heartbeater.cc:503] Master 127.10.11.254:34033 was elected leader, sending a full tablet report...
W20260927 00:33:37.675064 11049 tablet.cc:2369] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:37.833997 11340 raft_consensus.cc:482] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:37.834252 11340 raft_consensus.cc:504] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.835453 11340 leader_election.cc:288] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b3e5b547b4429eaf3ec4d42b0e4a38 (127.10.11.194:46187), b658186909084157946c3f2d70fbe56e (127.10.11.193:39707)
I20260927 00:33:37.840076 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.835758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:59901 (local address 127.10.11.194:46187)
0927 00:33:37.835947 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.835953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.835973 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:33:37.836033 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.836036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.836088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.836175 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:37.836487 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.837293 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.838854 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.839437 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.839546 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.839594 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.839921 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.839980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 00:33:37.840076 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.835773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:51169 (local address 127.10.11.193:39707)
0927 00:33:37.835913 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.835917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.836301 (+ 384us) server_negotiation.cc:407] Connection header received
0927 00:33:37.836390 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.836393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.836456 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.836535 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:37.836758 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.837459 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.838828 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.839427 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.839546 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.839594 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.839868 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.839953 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:33:37.840088 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.835661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:46187 (local address 127.10.11.195:59901)
0927 00:33:37.835823 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.835842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.835937 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.836205 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.836210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.836235 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:37.836367 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.836374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.837401 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.837403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.838685 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.838696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.839544 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.839548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.839747 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.839833 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.839900 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.839971 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":87}
I20260927 00:33:37.840097 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.835671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39707 (local address 127.10.11.195:51169)
0927 00:33:37.836204 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:37.836216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:37.836310 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:37.836548 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:37.836550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:37.836568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:37.836652 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.836656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.837546 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.837549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.838702 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:37.838708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.839544 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.839548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:37.839701 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.839770 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 00:33:37.839850 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:33:37.839944 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":204,"thread_start_us":137,"threads_started":1}
I20260927 00:33:37.840824 11002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359514f904864024b284a6b4b434222d" candidate_uuid: "c74de56ef61f4571a4c60974a8170842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b658186909084157946c3f2d70fbe56e" is_pre_election: true
I20260927 00:33:37.840835 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359514f904864024b284a6b4b434222d" candidate_uuid: "c74de56ef61f4571a4c60974a8170842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" is_pre_election: true
I20260927 00:33:37.841114 11136 raft_consensus.cc:2454] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74de56ef61f4571a4c60974a8170842 in term 0.
I20260927 00:33:37.841114 11002 raft_consensus.cc:2454] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74de56ef61f4571a4c60974a8170842 in term 0.
I20260927 00:33:37.841606 11205 leader_election.cc:302] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22b3e5b547b4429eaf3ec4d42b0e4a38, b658186909084157946c3f2d70fbe56e, c74de56ef61f4571a4c60974a8170842; no voters:
I20260927 00:33:37.841917 11340 raft_consensus.cc:2781] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:37.842006 11340 raft_consensus.cc:482] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:37.842067 11340 raft_consensus.cc:3037] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:37.843197 11340 raft_consensus.cc:504] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
I20260927 00:33:37.843529 11340 leader_election.cc:288] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [CANDIDATE]: Term 1 election: Requested vote from peers 22b3e5b547b4429eaf3ec4d42b0e4a38 (127.10.11.194:46187), b658186909084157946c3f2d70fbe56e (127.10.11.193:39707)
I20260927 00:33:37.843875 11002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359514f904864024b284a6b4b434222d" candidate_uuid: "c74de56ef61f4571a4c60974a8170842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b658186909084157946c3f2d70fbe56e"
I20260927 00:33:37.843875 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359514f904864024b284a6b4b434222d" candidate_uuid: "c74de56ef61f4571a4c60974a8170842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38"
I20260927 00:33:37.844002 11002 raft_consensus.cc:3037] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:37.844007 11136 raft_consensus.cc:3037] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:37.844944 11002 raft_consensus.cc:2454] T 359514f904864024b284a6b4b434222d P b658186909084157946c3f2d70fbe56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74de56ef61f4571a4c60974a8170842 in term 1.
I20260927 00:33:37.845093 11136 raft_consensus.cc:2454] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74de56ef61f4571a4c60974a8170842 in term 1.
I20260927 00:33:37.845232 11205 leader_election.cc:302] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b658186909084157946c3f2d70fbe56e, c74de56ef61f4571a4c60974a8170842; no voters:
I20260927 00:33:37.845386 11340 raft_consensus.cc:2781] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:37.845597 11340 raft_consensus.cc:686] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [term 1 LEADER]: Becoming Leader. State: Replica: c74de56ef61f4571a4c60974a8170842, State: Running, Role: LEADER
I20260927 00:33:37.845940 11340 consensus_queue.cc:237] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } }
W20260927 00:33:37.846463 11317 tablet.cc:2369] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:37.848942 10857 catalog_manager.cc:5495] T 359514f904864024b284a6b4b434222d P c74de56ef61f4571a4c60974a8170842 reported cstate change: term changed from 0 to 1, leader changed from <none> to c74de56ef61f4571a4c60974a8170842 (127.10.11.195). New cstate: current_term: 1 leader_uuid: "c74de56ef61f4571a4c60974a8170842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74de56ef61f4571a4c60974a8170842" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 34231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22b3e5b547b4429eaf3ec4d42b0e4a38" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b658186909084157946c3f2d70fbe56e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39707 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:37.885174 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.876535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.10.11.193:39707)
0927 00:33:37.876673 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.876678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.876847 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:33:37.876956 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.876976 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.877032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.877117 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.877899 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.878457 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.879249 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.879464 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.879545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.881972 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.881981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.881995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.882015 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.883628 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.884012 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.884018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.884019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.884106 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.884324 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.884327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.884328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.884552 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.884704 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.884878 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.885030 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:33:37.887679 11183 tablet.cc:2369] T 359514f904864024b284a6b4b434222d P 22b3e5b547b4429eaf3ec4d42b0e4a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:37.910600 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.902583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.10.11.193:39707)
0927 00:33:37.902705 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.902710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.902906 (+ 196us) server_negotiation.cc:407] Connection header received
0927 00:33:37.903052 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.903060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.903097 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.903177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.903804 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.904295 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.905089 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.905310 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.905414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.907706 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.907715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.907717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.907738 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.909006 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.909398 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.909403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.909405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.909479 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.909736 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.909740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.909742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.909897 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.910050 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.910336 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.910450 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:33:37 dist-test-slave-6gxc krb5kdc[10812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:37.934937 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:37.927071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.10.11.193:39707)
0927 00:33:37.927238 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:37.927241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:37.927309 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:33:37.927401 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:37.927403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:37.927448 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:37.927509 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:37.928180 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.928695 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.929578 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:37.929783 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:37.929877 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:37.932104 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:37.932107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:37.932109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:37.932128 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:37.933330 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.933754 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.933758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.933759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.933839 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:37.934126 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:37.934129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:37.934131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:37.934281 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:37.934413 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:37.934642 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:33:37.934767 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:33:37.936162 10962 tablet_service.cc:2908] Rejecting scan request for tablet 359514f904864024b284a6b4b434222d: Uninitialized: clean time has not yet been initialized
I20260927 00:33:37.938019 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10919
I20260927 00:33:37.943989 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11053
I20260927 00:33:37.949193 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11187
I20260927 00:33:37.954375 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 10825
2026-09-27T00:33:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2376 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:38 dist-test-slave-6gxc krb5kdc[11357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:38 dist-test-slave-6gxc krb5kdc[11357](info): set up 2 sockets
Sep 27 00:33:38 dist-test-slave-6gxc krb5kdc[11357](info): commencing operation
krb5kdc: starting...
W20260927 00:33:38.990401 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:39.101290 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:44249
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:39.206301 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:39.206555 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:39.206621 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:39.210505 11370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:39.210572 11370 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:33:39.210604 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:39.210657 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:39.210707 11370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:39.210747 11370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:39.215832 11370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:44249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:39.216715 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:39.217603 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:39.225315 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.225327 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.225322 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:39.225591 11370 server_base.cc:1034] running on GCE node
I20260927 00:33:39.226228 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:39.227164 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:39.228359 11370 hybrid_clock.cc:648] HybridClock initialized: now 1790469219228349 us; error 28 us; skew 500 ppm
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:39.230970 11370 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:39.231665 11370 webserver.cc:460] Webserver started at http://127.10.11.254:44077/ using document root <none> and password file <none>
I20260927 00:33:39.232120 11370 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:39.232172 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:39.232312 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:39.234053 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "0800c28bdf60471cb2336aefd56b7288"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "44ab0789aeb1a54f116367784bc1f411"
server_key_iv: "82ec47574f132f3872df17546ba89c0c"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.234594 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "0800c28bdf60471cb2336aefd56b7288"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "44ab0789aeb1a54f116367784bc1f411"
server_key_iv: "82ec47574f132f3872df17546ba89c0c"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.237800 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:39.239951 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.241060 11370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:39.241173 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "0800c28bdf60471cb2336aefd56b7288"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "44ab0789aeb1a54f116367784bc1f411"
server_key_iv: "82ec47574f132f3872df17546ba89c0c"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.241264 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:39.266603 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:39.269295 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:39.269474 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:39.276793 11370 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:44249
I20260927 00:33:39.276824 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:44249 every 8 connection(s)
I20260927 00:33:39.277621 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:39.280273 11448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:39.285315 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Bootstrap starting.
I20260927 00:33:39.286300 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11370
I20260927 00:33:39.286428 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:39.286652 10287 external_mini_cluster.cc:1428] Setting key 6e812da3849b8f653b494d5261ebde3b
I20260927 00:33:39.287880 11448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:39.288721 11448 log.cc:824] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:39.290717 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: No bootstrap required, opened a new log
I20260927 00:33:39.294728 11448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } }
I20260927 00:33:39.294930 11448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:39.295011 11448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0800c28bdf60471cb2336aefd56b7288, State: Initialized, Role: FOLLOWER
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:39.295425 11448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } }
I20260927 00:33:39.295596 11448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:39.295662 11448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:39.295758 11448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:39.296834 11448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } }
I20260927 00:33:39.297211 11448 leader_election.cc:302] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0800c28bdf60471cb2336aefd56b7288; no voters:
I20260927 00:33:39.297539 11448 leader_election.cc:288] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:39.297636 11453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:39.297842 11453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [term 1 LEADER]: Becoming Leader. State: Replica: 0800c28bdf60471cb2336aefd56b7288, State: Running, Role: LEADER
I20260927 00:33:39.298138 11453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } }
I20260927 00:33:39.298774 11448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:39.299628 11455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0800c28bdf60471cb2336aefd56b7288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } } }
I20260927 00:33:39.299743 11455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:39.300017 11454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0800c28bdf60471cb2336aefd56b7288. Latest consensus state: current_term: 1 leader_uuid: "0800c28bdf60471cb2336aefd56b7288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800c28bdf60471cb2336aefd56b7288" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44249 } } }
I20260927 00:33:39.300140 11454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:39.301975 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.287669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.10.11.254:44249)
0927 00:33:39.288110 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.288130 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.288183 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:33:39.288950 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.288973 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.289303 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.289636 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.290309 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.291721 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.292564 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.293279 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.293438 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.296043 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.296049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.296064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.296093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.298132 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.298634 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.298639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.298644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.298780 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.299259 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.299263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.299265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.299805 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.300023 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.300892 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.301179 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
W20260927 00:33:39.303454 11469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:39.303527 11469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:39.303581 11462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:39.304359 11462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:39.309448 11462 catalog_manager.cc:1347] Generated new cluster ID: 0c8103df9b0a42aabf1a68a2ef85e739
I20260927 00:33:39.309512 11462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:39.318284 11462 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:39.319077 11462 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:39.330785 11462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Generated new TSK 0
I20260927 00:33:39.331396 11462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:39.365075 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44249
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:39.486708 11474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:39.486922 11474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:39.486989 11474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:39.490909 11474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:39.490954 11474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:39.490991 11474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:39.491066 11474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:39.495970 11474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:39.496857 11474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:39.497749 11474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:39.507076 11482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.507102 11480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.507130 11479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:39.507313 11474 server_base.cc:1034] running on GCE node
I20260927 00:33:39.507683 11474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:39.508265 11474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:39.509441 11474 hybrid_clock.cc:648] HybridClock initialized: now 1790469219509427 us; error 27 us; skew 500 ppm
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:39.512141 11474 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:39.512867 11474 webserver.cc:460] Webserver started at http://127.10.11.193:39265/ using document root <none> and password file <none>
I20260927 00:33:39.513334 11474 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:39.513389 11474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:39.513533 11474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:39.515398 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "2d40cd73dd1843c6bf99220b6d4e5583"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "27e3c60a52bd9e3821711e414ad550fe"
server_key_iv: "a851cd4231e2963305ee96d86b4a6e34"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.515895 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "2d40cd73dd1843c6bf99220b6d4e5583"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "27e3c60a52bd9e3821711e414ad550fe"
server_key_iv: "a851cd4231e2963305ee96d86b4a6e34"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.519228 11474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:39.521481 11489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.522783 11474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:39.522884 11474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "2d40cd73dd1843c6bf99220b6d4e5583"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "27e3c60a52bd9e3821711e414ad550fe"
server_key_iv: "a851cd4231e2963305ee96d86b4a6e34"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.522962 11474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:39.541119 11474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:39.544085 11474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:39.544220 11474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:39.545614 11474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:39.545662 11474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.545723 11474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:39.545756 11474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:39.556615 11474 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44437
I20260927 00:33:39.556738 11604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44437 every 8 connection(s)
I20260927 00:33:39.557480 11474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:39.560248 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.546288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:42581 (local address 127.10.11.254:44249)
0927 00:33:39.546407 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.546411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.547349 (+ 938us) server_negotiation.cc:407] Connection header received
0927 00:33:39.548300 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.548304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.548361 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.548454 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.549932 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.550859 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.552782 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.553239 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.553348 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.557101 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.557104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.557106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.557128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.558660 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.559082 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.559084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.559085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.559159 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.559395 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.559397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.559398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.559550 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.559622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.560035 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.560123 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:33:39.560796 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11474
I20260927 00:33:39.560909 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:39.561054 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.546773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.193:42581)
0927 00:33:39.547215 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.547256 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.548102 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.548621 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.548631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.549121 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.549754 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.549773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.551027 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.551032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.552621 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.552637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.553514 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.553521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.553753 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.554970 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.554994 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.556901 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.558758 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.558762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.558771 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.559001 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.559219 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.559220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.559222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.559327 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.559632 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.559636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.559851 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.559993 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.560203 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260927 00:33:39.561201 10287 external_mini_cluster.cc:1428] Setting key 0dc9ec207897b4120b5b346b60ff7ad4
I20260927 00:33:39.562187 11605 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44249
I20260927 00:33:39.564126 11605 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:39.564569 11605 heartbeater.cc:511] Master 127.10.11.254:44249 requested a full tablet report, sending...
I20260927 00:33:39.565951 11401 ts_manager.cc:194] Registered new tserver with Master: 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193:44437)
I20260927 00:33:39.567461 11401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:42581
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:33:39.571892 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.564257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:46017 (local address 127.10.11.254:44249)
0927 00:33:39.564388 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.564395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.564764 (+ 369us) server_negotiation.cc:407] Connection header received
0927 00:33:39.564936 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.564942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.564992 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.565048 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.565561 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.566287 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.567428 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.567843 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.567977 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.568942 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.568946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.568948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.568970 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.570347 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.570757 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.570761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.570762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.570835 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.571101 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.571104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.571106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.571297 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.571401 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.571644 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.571755 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:33:39.571900 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.564366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.193:46017)
0927 00:33:39.564681 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.564699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.564834 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.565197 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.565203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.565378 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.565460 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.565465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.566431 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.566435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.567294 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.567305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.567980 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.567986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.568087 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.568526 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.568537 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.568817 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.570464 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.570468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.570469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.570659 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.570928 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.570930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.570932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.571011 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.571402 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.571406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.571495 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.571595 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.571721 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:39.587208 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44249
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:39.691439 11614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:39.691692 11614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:39.691761 11614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:39.695717 11614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:39.695785 11614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:39.695851 11614 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:39.695926 11614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:39.701066 11614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:39.701963 11614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:39.702920 11614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:39.711647 11620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.711763 11619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.711651 11622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:39.711939 11614 server_base.cc:1034] running on GCE node
I20260927 00:33:39.712260 11614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:39.712813 11614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:39.713974 11614 hybrid_clock.cc:648] HybridClock initialized: now 1790469219713942 us; error 42 us; skew 500 ppm
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:39.716598 11614 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:39.717331 11614 webserver.cc:460] Webserver started at http://127.10.11.194:35239/ using document root <none> and password file <none>
I20260927 00:33:39.717793 11614 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:39.717846 11614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:39.717993 11614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:39.719836 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "3756bdd2e20a408aaeee5c735f87409f"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "db69cce5232fc2b9d273d4d52dc498b9"
server_key_iv: "c6fbfb5e720958ae30b5d0c888c93883"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.720314 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "3756bdd2e20a408aaeee5c735f87409f"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "db69cce5232fc2b9d273d4d52dc498b9"
server_key_iv: "c6fbfb5e720958ae30b5d0c888c93883"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.723672 11614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:33:39.725870 11629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.727056 11614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:39.727164 11614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "3756bdd2e20a408aaeee5c735f87409f"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "db69cce5232fc2b9d273d4d52dc498b9"
server_key_iv: "c6fbfb5e720958ae30b5d0c888c93883"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.727253 11614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:39.747745 11614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:39.750732 11614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:39.750916 11614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:39.752262 11614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:39.752328 11614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.752391 11614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:39.752431 11614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:39.763098 11614 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:42457
I20260927 00:33:39.763396 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:42457 every 8 connection(s)
I20260927 00:33:39.764125 11614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:39.765846 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.753064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:33727 (local address 127.10.11.254:44249)
0927 00:33:39.753189 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.753192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.754141 (+ 949us) server_negotiation.cc:407] Connection header received
0927 00:33:39.755247 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.755254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.755317 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.755381 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.757015 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.757809 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.759160 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.759393 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.759506 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.762322 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.762325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.762327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.762352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.763898 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.764457 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.764462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.764463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.764554 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.764925 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.764929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.764930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.765140 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.765219 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.765640 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.765729 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:33:39.766606 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.753541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.194:33727)
0927 00:33:39.753979 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.754018 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.755007 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.755800 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.755810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.756267 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.756875 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.756894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.757919 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.757925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.759045 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.759059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.759546 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.759550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.759754 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.760364 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.760380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.762144 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.764008 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.764013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.764025 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.764344 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.764661 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.764664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.764666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.764813 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.765239 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.765243 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.765437 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.765577 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.765805 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260927 00:33:39.767603 11745 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44249
I20260927 00:33:39.767735 11745 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:39.768199 11745 heartbeater.cc:511] Master 127.10.11.254:44249 requested a full tablet report, sending...
I20260927 00:33:39.769131 11402 ts_manager.cc:194] Registered new tserver with Master: 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457)
I20260927 00:33:39.769833 11402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:33727
I20260927 00:33:39.772482 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11614
I20260927 00:33:39.772608 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:39.772868 10287 external_mini_cluster.cc:1428] Setting key f143e6cf0905e893f859feff07eeb293
I20260927 00:33:39.777361 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.769591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:49073 (local address 127.10.11.254:44249)
0927 00:33:39.769701 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.769704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.770333 (+ 629us) server_negotiation.cc:407] Connection header received
0927 00:33:39.770457 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.770460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.770504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.770597 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.771007 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.771749 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.772704 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.772961 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.773058 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.774247 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.774251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.774254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.774277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.775571 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.776104 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.776107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.776108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.776181 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.776488 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.776491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.776492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.776733 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.776853 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.777150 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.777247 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:33:39.777560 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.769498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.194:49073)
0927 00:33:39.770211 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.770262 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.770360 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.770606 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.770609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.770770 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.770896 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.770901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.771867 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.771869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.772534 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.772546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.773082 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.773085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.773191 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.773690 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.773702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.774077 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.775722 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.775725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.775728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.775991 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.776271 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.776275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.776279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.776378 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.776853 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.776857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.776971 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.777084 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.777260 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:39.798274 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44249
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:39.903285 11754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:39.903491 11754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:39.903530 11754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:39.907469 11754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:39.907518 11754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:39.907574 11754 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:39.907641 11754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:39.912629 11754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:39.913486 11754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:39.914386 11754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:39.923296 11760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.923367 11759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:39.923296 11762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:39.923451 11754 server_base.cc:1034] running on GCE node
I20260927 00:33:39.923854 11754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:39.924451 11754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:39.925634 11754 hybrid_clock.cc:648] HybridClock initialized: now 1790469219925624 us; error 29 us; skew 500 ppm
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:39.928421 11754 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:39.929188 11754 webserver.cc:460] Webserver started at http://127.10.11.195:40113/ using document root <none> and password file <none>
I20260927 00:33:39.929687 11754 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:39.929780 11754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:39.929983 11754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:39.931896 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "a18711cb23c5434c9008022a888cbc10"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "e782d4bc5161e93f0b98c8c5ba81a9ab"
server_key_iv: "9259bd9451714c23fb5be671915779c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.932471 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "a18711cb23c5434c9008022a888cbc10"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "e782d4bc5161e93f0b98c8c5ba81a9ab"
server_key_iv: "9259bd9451714c23fb5be671915779c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.935950 11754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:39.938381 11769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.939509 11754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:39.939654 11754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "a18711cb23c5434c9008022a888cbc10"
format_stamp: "Formatted at 2026-09-27 00:33:39 on dist-test-slave-6gxc"
server_key: "e782d4bc5161e93f0b98c8c5ba81a9ab"
server_key_iv: "9259bd9451714c23fb5be671915779c8"
server_key_version: "encryptionkey@0"
I20260927 00:33:39.939795 11754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:39.953045 11754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:39.956218 11754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:39.956414 11754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:39.957813 11754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:39.957886 11754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:39.957955 11754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:39.958010 11754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:39.969511 11754 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:43177
I20260927 00:33:39.969571 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:43177 every 8 connection(s)
I20260927 00:33:39.970443 11754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:39.971989 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.958464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:36689 (local address 127.10.11.254:44249)
0927 00:33:39.958617 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.958624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.959945 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 00:33:39.961292 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.961296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.961356 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.961464 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.962711 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.963354 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.965141 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.965441 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.965519 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.968354 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.968358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.968360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.968382 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.969914 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.970451 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.970456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.970458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.970559 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.970930 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.970934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.970935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.971103 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.971204 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.971704 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.971828 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:33:39.972663 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.959284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.195:36689)
0927 00:33:39.959772 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.959825 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.961022 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.961628 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.961634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.961999 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.962566 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.962582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.963492 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.963500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.965002 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.965017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.965602 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.965610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.965838 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.966456 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.966483 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.968181 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.970030 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.970035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.970048 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.970351 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.970655 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.970660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.970662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.970823 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.971202 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.971209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.971473 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.971643 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.971927 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260927 00:33:39.973876 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11754
I20260927 00:33:39.973876 11885 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44249
I20260927 00:33:39.973994 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:39.974016 11885 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:39.974336 10287 external_mini_cluster.cc:1428] Setting key cda8fe967b4bc31521b2e2ef90ab8381
I20260927 00:33:39.974545 11885 heartbeater.cc:511] Master 127.10.11.254:44249 requested a full tablet report, sending...
I20260927 00:33:39.975790 11402 ts_manager.cc:194] Registered new tserver with Master: a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177)
I20260927 00:33:39.976533 11402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:36689
I20260927 00:33:39.982751 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.976126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.10.11.195:47275)
0927 00:33:39.976396 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:39.976412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:39.976512 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:39.976782 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:39.976785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:39.977003 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:39.977149 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.977157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.977885 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.977889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.978651 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:39.978660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.979034 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.979036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:39.979104 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.979542 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:39.979554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:39.979862 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:39.981462 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.981464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.981466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.981654 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.981929 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:39.981931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:39.981933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:39.982021 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:39.982371 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:39.982373 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:39.982443 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:33:39.982525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:33:39.982629 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260927 00:33:39.982764 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:39.975995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:47275 (local address 127.10.11.254:44249)
0927 00:33:39.976187 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:39.976193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:39.976486 (+ 293us) server_negotiation.cc:407] Connection header received
0927 00:33:39.976600 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:39.976602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:39.976665 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:39.976746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:39.977286 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.977760 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.978748 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:39.978942 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:39.979015 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:39.979989 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:39.979992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:39.979994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:39.980014 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:39.981364 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.981764 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.981766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.981768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.981835 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:39.982134 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:39.982139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:39.982141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:39.982293 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:39.982366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:39.982553 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:33:39.982634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 00:33:39.987974 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:33:39 dist-test-slave-6gxc krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:40.012072 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.004371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.10.11.254:44249)
0927 00:33:40.004490 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.004493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.004505 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:33:40.004539 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.004541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.004576 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.004641 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:40.005276 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.005815 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.006651 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.006878 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.006958 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.009224 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:40.009227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:40.009229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:40.009253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:40.010623 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.011001 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.011004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.011005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.011084 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.011303 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.011305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.011307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.011468 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:40.011573 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.011757 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.011848 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:33:40.014726 11402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:40.018141 11402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:40.032220 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.026892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.10.11.195:43177)
0927 00:33:40.027858 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.027874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.027899 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:33:40.027982 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.027988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.028216 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.028394 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.028606 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.029591 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.031039 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.031570 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.031699 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.031861 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.031992 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.032080 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":117,"threads_started":1}
I20260927 00:33:40.032212 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.026243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:43177 (local address 127.0.0.1:50644)
0927 00:33:40.027336 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.027389 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.027535 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.028379 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.028383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.028400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.028493 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.028497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.029687 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.029691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.030941 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.030952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.031688 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.031692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.031810 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.031851 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.031964 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.032074 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":78,"threads_started":1}
I20260927 00:33:40.033192 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.027469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44437 (local address 127.0.0.1:33448)
0927 00:33:40.027982 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.027996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.028076 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.028810 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.028813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.028831 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.028923 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.028929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.030449 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.030454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.031735 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.031741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.032820 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.032823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.032910 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.032933 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.032995 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.033053 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":35,"thread_start_us":91,"threads_started":1}
I20260927 00:33:40.033289 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.027978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.10.11.193:44437)
0927 00:33:40.028349 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.028365 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.028387 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:33:40.028460 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.028465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.028643 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.028810 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.029022 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.030307 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.031883 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.032706 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.032819 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.032972 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.033029 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.033116 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":86,"threads_started":1}
I20260927 00:33:40.033776 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.027485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.10.11.194:42457)
0927 00:33:40.028683 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.028697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.028723 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:33:40.028799 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.028804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.028962 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.029127 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.029286 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.030383 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.031883 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.032702 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.032819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.032972 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.033576 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.033649 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":169,"threads_started":1}
I20260927 00:33:40.033484 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.026737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:42457 (local address 127.0.0.1:39588)
0927 00:33:40.027746 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.028140 (+ 394us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.028212 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.029092 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.029095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.029109 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.029189 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.029194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.030500 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.030505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.031728 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.031740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.033039 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.033043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.033132 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.033158 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.033248 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.033330 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":37,"thread_start_us":103,"threads_started":1}
I20260927 00:33:40.035403 11819 tablet_service.cc:1467] Processing CreateTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 (DEFAULT_TABLE table=test-table [id=208abdaaae724dfe8fb3242c2a9321d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.036497 11819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f66495691a74f178d09faf1a3bab5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.036747 11679 tablet_service.cc:1467] Processing CreateTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 (DEFAULT_TABLE table=test-table [id=208abdaaae724dfe8fb3242c2a9321d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.037674 11679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f66495691a74f178d09faf1a3bab5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.039536 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 (DEFAULT_TABLE table=test-table [id=208abdaaae724dfe8fb3242c2a9321d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.040267 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f66495691a74f178d09faf1a3bab5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.044713 11908 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Bootstrap starting.
I20260927 00:33:40.046492 11909 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Bootstrap starting.
I20260927 00:33:40.046518 11910 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Bootstrap starting.
I20260927 00:33:40.047305 11908 tablet_bootstrap.cc:654] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.048321 11908 log.cc:824] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:40.048547 11909 tablet_bootstrap.cc:654] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.048547 11910 tablet_bootstrap.cc:654] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.049331 11910 log.cc:824] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:40.049331 11909 log.cc:824] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:40.050179 11908 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: No bootstrap required, opened a new log
I20260927 00:33:40.050375 11908 ts_tablet_manager.cc:1397] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:33:40.051271 11910 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: No bootstrap required, opened a new log
I20260927 00:33:40.051447 11910 ts_tablet_manager.cc:1397] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:33:40.051508 11909 tablet_bootstrap.cc:492] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: No bootstrap required, opened a new log
I20260927 00:33:40.051657 11909 ts_tablet_manager.cc:1397] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:33:40.053258 11908 raft_consensus.cc:348] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.053493 11908 raft_consensus.cc:374] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.053548 11908 raft_consensus.cc:729] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18711cb23c5434c9008022a888cbc10, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.053953 11908 consensus_queue.cc:260] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.055059 11908 ts_tablet_manager.cc:1428] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:33:40.055151 11885 heartbeater.cc:503] Master 127.10.11.254:44249 was elected leader, sending a full tablet report...
I20260927 00:33:40.055249 11909 raft_consensus.cc:348] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.055483 11909 raft_consensus.cc:374] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.055539 11909 raft_consensus.cc:729] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3756bdd2e20a408aaeee5c735f87409f, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.055794 11910 raft_consensus.cc:348] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.055929 11909 consensus_queue.cc:260] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.056006 11910 raft_consensus.cc:374] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.056073 11910 raft_consensus.cc:729] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d40cd73dd1843c6bf99220b6d4e5583, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.056540 11745 heartbeater.cc:503] Master 127.10.11.254:44249 was elected leader, sending a full tablet report...
I20260927 00:33:40.056497 11910 consensus_queue.cc:260] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.056829 11909 ts_tablet_manager.cc:1428] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:33:40.057351 11910 ts_tablet_manager.cc:1428] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 00:33:40.057643 11605 heartbeater.cc:503] Master 127.10.11.254:44249 was elected leader, sending a full tablet report...
W20260927 00:33:40.058658 11606 tablet.cc:2369] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:33:40.221431 11886 tablet.cc:2369] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:33:40.265306 11746 tablet.cc:2369] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:40.303826 11915 raft_consensus.cc:482] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:40.304052 11915 raft_consensus.cc:504] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.305243 11915 leader_election.cc:288] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177), 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193:44437)
I20260927 00:33:40.309010 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.305518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44437 (local address 127.10.11.194:33831)
0927 00:33:40.305856 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.305868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.305981 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.306273 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.306276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.306292 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.306407 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.306412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.307098 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.307105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.307948 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.307957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.308559 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.308666 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.308691 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.308778 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.308848 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260927 00:33:40.309031 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.305621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:33831 (local address 127.10.11.193:44437)
0927 00:33:40.305751 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.305756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.305945 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:33:40.306094 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.306132 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.306179 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.306252 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.306511 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.306999 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308066 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.308450 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308571 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.308624 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.308813 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.308901 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:33:40.309051 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.305430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:43177 (local address 127.10.11.194:48165)
0927 00:33:40.305754 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.305777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.305916 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.306225 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.306229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.306245 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.306368 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.306374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.307112 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.307119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.307958 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.307968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.308694 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.308797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.308816 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.308880 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.308935 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":46,"thread_start_us":73,"threads_started":1}
I20260927 00:33:40.309084 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.305575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:48165 (local address 127.10.11.195:43177)
0927 00:33:40.305728 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.305734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.305853 (+ 119us) server_negotiation.cc:407] Connection header received
0927 00:33:40.306025 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.306028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.306079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.306189 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.306476 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.307000 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308095 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.308561 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.308719 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.308799 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.308900 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.308968 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:33:40.309751 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f66495691a74f178d09faf1a3bab5d7" candidate_uuid: "3756bdd2e20a408aaeee5c735f87409f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10" is_pre_election: true
I20260927 00:33:40.309823 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f66495691a74f178d09faf1a3bab5d7" candidate_uuid: "3756bdd2e20a408aaeee5c735f87409f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" is_pre_election: true
I20260927 00:33:40.310051 11839 raft_consensus.cc:2454] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3756bdd2e20a408aaeee5c735f87409f in term 0.
I20260927 00:33:40.310140 11559 raft_consensus.cc:2454] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3756bdd2e20a408aaeee5c735f87409f in term 0.
I20260927 00:33:40.310600 11632 leader_election.cc:302] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, 3756bdd2e20a408aaeee5c735f87409f; no voters:
I20260927 00:33:40.310863 11915 raft_consensus.cc:2781] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:40.310941 11915 raft_consensus.cc:482] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:40.310998 11915 raft_consensus.cc:3037] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.312252 11915 raft_consensus.cc:504] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.312588 11915 leader_election.cc:288] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [CANDIDATE]: Term 1 election: Requested vote from peers a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177), 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193:44437)
I20260927 00:33:40.312913 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f66495691a74f178d09faf1a3bab5d7" candidate_uuid: "3756bdd2e20a408aaeee5c735f87409f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10"
I20260927 00:33:40.312950 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f66495691a74f178d09faf1a3bab5d7" candidate_uuid: "3756bdd2e20a408aaeee5c735f87409f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d40cd73dd1843c6bf99220b6d4e5583"
I20260927 00:33:40.313051 11839 raft_consensus.cc:3037] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.313072 11559 raft_consensus.cc:3037] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.314049 11559 raft_consensus.cc:2454] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3756bdd2e20a408aaeee5c735f87409f in term 1.
I20260927 00:33:40.314371 11632 leader_election.cc:302] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, 3756bdd2e20a408aaeee5c735f87409f; no voters:
I20260927 00:33:40.314519 11915 raft_consensus.cc:2781] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:40.314683 11915 raft_consensus.cc:686] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 1 LEADER]: Becoming Leader. State: Replica: 3756bdd2e20a408aaeee5c735f87409f, State: Running, Role: LEADER
I20260927 00:33:40.314769 11839 raft_consensus.cc:2454] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3756bdd2e20a408aaeee5c735f87409f in term 1.
I20260927 00:33:40.315003 11915 consensus_queue.cc:237] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } }
I20260927 00:33:40.317836 11400 catalog_manager.cc:5495] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194). New cstate: current_term: 1 leader_uuid: "3756bdd2e20a408aaeee5c735f87409f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } health_report { overall_health: HEALTHY } } }
I20260927 00:33:40.395030 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.391198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.10.11.194:42457)
0927 00:33:40.391350 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.391355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.391371 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:33:40.391589 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.391592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.391648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.391761 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:40.392043 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.392562 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.393498 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.393728 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.393841 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.394226 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.394590 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.394720 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.394822 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.394879 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:33:40.400586 11559 raft_consensus.cc:1261] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Refusing update from remote peer 3756bdd2e20a408aaeee5c735f87409f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.400640 11839 raft_consensus.cc:1261] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Refusing update from remote peer 3756bdd2e20a408aaeee5c735f87409f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.401196 11915 consensus_queue.cc:1035] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.401397 11919 consensus_queue.cc:1035] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.410815 11923 mvcc.cc:205] Tried to move back new op lower bound from 7333761926753320960 to 7333761926412828672. Current Snapshot: MvccSnapshot[applied={T|T < 7333761926753320960 or (T in {7333761926753320960})}]
I20260927 00:33:40.419427 11401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:40.419577 11401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:40.421754 11401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Sending DeleteTablet for 3 replicas of tablet 0f66495691a74f178d09faf1a3bab5d7
I20260927 00:33:40.422425 11819 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:50644
I20260927 00:33:40.422429 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:33448
I20260927 00:33:40.422559 11679 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f66495691a74f178d09faf1a3bab5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:39588
I20260927 00:33:40.422958 11933 tablet_replica.cc:330] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: stopping tablet replica
I20260927 00:33:40.423115 11932 tablet_replica.cc:330] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: stopping tablet replica
I20260927 00:33:40.423242 11933 raft_consensus.cc:2229] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:40.423339 11932 raft_consensus.cc:2229] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:40.423493 11933 raft_consensus.cc:2258] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.423547 11932 raft_consensus.cc:2258] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.423660 11401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:33:40.423820 11934 tablet_replica.cc:330] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: stopping tablet replica
W20260927 00:33:40.424026 11401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:40.424074 11934 raft_consensus.cc:2229] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:40.424420 11934 raft_consensus.cc:2258] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.424973 11933 ts_tablet_manager.cc:1905] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.426316 11934 ts_tablet_manager.cc:1905] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.428391 11539 tablet_service.cc:1467] Processing CreateTablet for tablet d97f99270a5545b394b8cb012ed682f0 (DEFAULT_TABLE table=test-table [id=20f39b0eddda40b29a571109b4a351e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.428633 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97f99270a5545b394b8cb012ed682f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.428617 11679 tablet_service.cc:1467] Processing CreateTablet for tablet d97f99270a5545b394b8cb012ed682f0 (DEFAULT_TABLE table=test-table [id=20f39b0eddda40b29a571109b4a351e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.428838 11679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97f99270a5545b394b8cb012ed682f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.430935 11909 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Bootstrap starting.
I20260927 00:33:40.430982 11910 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Bootstrap starting.
I20260927 00:33:40.431412 11819 tablet_service.cc:1467] Processing CreateTablet for tablet d97f99270a5545b394b8cb012ed682f0 (DEFAULT_TABLE table=test-table [id=20f39b0eddda40b29a571109b4a351e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:40.431679 11819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97f99270a5545b394b8cb012ed682f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.431926 11909 tablet_bootstrap.cc:654] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.431929 11910 tablet_bootstrap.cc:654] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.432945 11909 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: No bootstrap required, opened a new log
I20260927 00:33:40.433041 11909 ts_tablet_manager.cc:1397] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:40.433431 11909 raft_consensus.cc:348] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.433473 11934 ts_tablet_manager.cc:1918] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:40.433547 11909 raft_consensus.cc:374] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.433585 11909 raft_consensus.cc:729] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3756bdd2e20a408aaeee5c735f87409f, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.433744 11909 consensus_queue.cc:260] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.434010 11909 ts_tablet_manager.cc:1428] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.434170 11908 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Bootstrap starting.
I20260927 00:33:40.434875 11932 ts_tablet_manager.cc:1905] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.435041 11908 tablet_bootstrap.cc:654] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.433581 11934 log.cc:1196] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/0f66495691a74f178d09faf1a3bab5d7
I20260927 00:33:40.436138 11934 ts_tablet_manager.cc:1939] T 0f66495691a74f178d09faf1a3bab5d7 P 3756bdd2e20a408aaeee5c735f87409f: Deleting consensus metadata
I20260927 00:33:40.437194 11932 ts_tablet_manager.cc:1918] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:40.437188 11386 catalog_manager.cc:4841] TS 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457): tablet 0f66495691a74f178d09faf1a3bab5d7 (table test-table [id=208abdaaae724dfe8fb3242c2a9321d0]) successfully deleted
I20260927 00:33:40.437280 11932 log.cc:1196] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/0f66495691a74f178d09faf1a3bab5d7
I20260927 00:33:40.437593 11932 ts_tablet_manager.cc:1939] T 0f66495691a74f178d09faf1a3bab5d7 P a18711cb23c5434c9008022a888cbc10: Deleting consensus metadata
I20260927 00:33:40.437961 11910 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: No bootstrap required, opened a new log
I20260927 00:33:40.438047 11910 ts_tablet_manager.cc:1397] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260927 00:33:40.438722 11387 catalog_manager.cc:4841] TS a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177): tablet 0f66495691a74f178d09faf1a3bab5d7 (table test-table [id=208abdaaae724dfe8fb3242c2a9321d0]) successfully deleted
I20260927 00:33:40.438717 11910 raft_consensus.cc:348] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.438815 11910 raft_consensus.cc:374] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.438848 11910 raft_consensus.cc:729] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d40cd73dd1843c6bf99220b6d4e5583, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.438972 11910 consensus_queue.cc:260] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.439247 11910 ts_tablet_manager.cc:1428] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.440325 11908 tablet_bootstrap.cc:492] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: No bootstrap required, opened a new log
I20260927 00:33:40.440407 11908 ts_tablet_manager.cc:1397] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 00:33:40.440685 11933 ts_tablet_manager.cc:1918] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:40.440783 11933 log.cc:1196] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/0f66495691a74f178d09faf1a3bab5d7
I20260927 00:33:40.440863 11908 raft_consensus.cc:348] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.440984 11908 raft_consensus.cc:374] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.441018 11908 raft_consensus.cc:729] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18711cb23c5434c9008022a888cbc10, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.441097 11933 ts_tablet_manager.cc:1939] T 0f66495691a74f178d09faf1a3bab5d7 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting consensus metadata
I20260927 00:33:40.441148 11908 consensus_queue.cc:260] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.441413 11908 ts_tablet_manager.cc:1428] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.442135 11388 catalog_manager.cc:4841] TS 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193:44437): tablet 0f66495691a74f178d09faf1a3bab5d7 (table test-table [id=208abdaaae724dfe8fb3242c2a9321d0]) successfully deleted
I20260927 00:33:40.569591 11916 raft_consensus.cc:482] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:40.569722 11916 raft_consensus.cc:504] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.570863 11916 leader_election.cc:288] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457), a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177)
I20260927 00:33:40.576131 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.571117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:38363 (local address 127.10.11.194:42457)
0927 00:33:40.571366 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.571370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.571421 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:33:40.571585 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.571589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.571638 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.571722 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.572093 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.572865 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.574818 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.575414 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.575533 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.575583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.575943 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.576010 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260927 00:33:40.576131 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.571186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:59195 (local address 127.10.11.195:43177)
0927 00:33:40.571315 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.571320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.571506 (+ 186us) server_negotiation.cc:407] Connection header received
0927 00:33:40.571588 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.571591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.571638 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.571722 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:40.572093 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.572882 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.574830 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.575411 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.575533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.575583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.575946 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.576010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:33:40.576211 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.571077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:43177 (local address 127.10.11.193:59195)
0927 00:33:40.571414 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.571427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.571509 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.571743 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.571745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.571782 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.571992 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.571997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.572982 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.572987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.574682 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.574693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.575526 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.575529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.575776 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.575839 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.575925 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.576035 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":364,"spinlock_wait_cycles":28032,"thread_start_us":86,"threads_started":1}
I20260927 00:33:40.576445 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.571058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:42457 (local address 127.10.11.193:38363)
0927 00:33:40.571326 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.571341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.571471 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.571743 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.571745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.571783 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:40.571984 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.571990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.572997 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.573002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.574682 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.574693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.575526 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.575529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.575780 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.575843 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.575919 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.576035 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":297,"spinlock_wait_cycles":5120,"thread_start_us":65,"threads_started":1}
I20260927 00:33:40.576639 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97f99270a5545b394b8cb012ed682f0" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10" is_pre_election: true
I20260927 00:33:40.576797 11839 raft_consensus.cc:2454] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 0.
I20260927 00:33:40.576984 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97f99270a5545b394b8cb012ed682f0" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3756bdd2e20a408aaeee5c735f87409f" is_pre_election: true
I20260927 00:33:40.577196 11699 raft_consensus.cc:2454] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 0.
I20260927 00:33:40.577185 11491 leader_election.cc:302] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, a18711cb23c5434c9008022a888cbc10; no voters:
I20260927 00:33:40.577422 11916 raft_consensus.cc:2781] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:40.577484 11916 raft_consensus.cc:482] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:40.577522 11916 raft_consensus.cc:3037] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.578480 11916 raft_consensus.cc:504] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.578751 11916 leader_election.cc:288] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 election: Requested vote from peers 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457), a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177)
I20260927 00:33:40.579069 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97f99270a5545b394b8cb012ed682f0" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3756bdd2e20a408aaeee5c735f87409f"
I20260927 00:33:40.579099 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97f99270a5545b394b8cb012ed682f0" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10"
I20260927 00:33:40.579183 11699 raft_consensus.cc:3037] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.579213 11839 raft_consensus.cc:3037] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.580015 11699 raft_consensus.cc:2454] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 1.
I20260927 00:33:40.580077 11839 raft_consensus.cc:2454] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 1.
I20260927 00:33:40.580322 11490 leader_election.cc:302] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, 3756bdd2e20a408aaeee5c735f87409f; no voters:
I20260927 00:33:40.580518 11916 raft_consensus.cc:2781] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:40.580711 11916 raft_consensus.cc:686] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 LEADER]: Becoming Leader. State: Replica: 2d40cd73dd1843c6bf99220b6d4e5583, State: Running, Role: LEADER
I20260927 00:33:40.580932 11916 consensus_queue.cc:237] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.582890 11401 catalog_manager.cc:5495] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:40.643499 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.635752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.10.11.254:44249)
0927 00:33:40.636009 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.636016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.636104 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:33:40.636183 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.636185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.636241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.636330 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:40.636963 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.637482 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.638311 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.638523 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.638599 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.640737 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:40.640740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:40.640742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:40.640761 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:40.641984 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.642482 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.642485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.642486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.642557 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.642786 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.642788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.642789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.642939 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:40.643010 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.643248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.643382 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20260927 00:33:40.643754 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.635672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44249 (local address 127.0.0.1:51164)
0927 00:33:40.635990 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.636004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.636113 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.636389 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.636391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.636766 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:40.636863 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.636869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.637577 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.637580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.638194 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.638202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.638615 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.638617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.638677 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.639199 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:40.639210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:40.640606 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:40.642080 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:40.642088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:40.642093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:40.642382 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:40.642633 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:40.642636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:40.642638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:40.642714 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:40.643021 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:40.643026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:40.643113 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.643235 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.643487 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":88,"thread_start_us":120,"threads_started":1}
I20260927 00:33:40.647202 11401 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:51164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:33:40.647910 11401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:40.652304 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 51afda3f8c0443f59d100a99c378f131 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba33d93d50544017863a4bf688492d11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:40.652305 11679 tablet_service.cc:1467] Processing CreateTablet for tablet 51afda3f8c0443f59d100a99c378f131 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba33d93d50544017863a4bf688492d11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:40.652390 11819 tablet_service.cc:1467] Processing CreateTablet for tablet 51afda3f8c0443f59d100a99c378f131 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba33d93d50544017863a4bf688492d11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:40.652685 11679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51afda3f8c0443f59d100a99c378f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.652685 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51afda3f8c0443f59d100a99c378f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.652777 11819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51afda3f8c0443f59d100a99c378f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:40.655259 11909 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: Bootstrap starting.
I20260927 00:33:40.655364 11910 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: Bootstrap starting.
I20260927 00:33:40.655370 11908 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: Bootstrap starting.
I20260927 00:33:40.655889 11909 tablet_bootstrap.cc:654] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.656236 11908 tablet_bootstrap.cc:654] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.656236 11910 tablet_bootstrap.cc:654] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:40.656803 11909 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: No bootstrap required, opened a new log
I20260927 00:33:40.656921 11909 ts_tablet_manager.cc:1397] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:40.657299 11909 raft_consensus.cc:348] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.657408 11909 raft_consensus.cc:374] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.657449 11909 raft_consensus.cc:729] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3756bdd2e20a408aaeee5c735f87409f, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.657603 11908 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: No bootstrap required, opened a new log
I20260927 00:33:40.657581 11909 consensus_queue.cc:260] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.657691 11908 ts_tablet_manager.cc:1397] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:40.657743 11910 tablet_bootstrap.cc:492] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: No bootstrap required, opened a new log
I20260927 00:33:40.657827 11910 ts_tablet_manager.cc:1397] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:33:40.657851 11909 ts_tablet_manager.cc:1428] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.658049 11927 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.658217 11927 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.658179 11908 raft_consensus.cc:348] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.658293 11908 raft_consensus.cc:374] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.658344 11908 raft_consensus.cc:729] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18711cb23c5434c9008022a888cbc10, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.658392 11910 raft_consensus.cc:348] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.658494 11910 raft_consensus.cc:374] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:40.658468 11908 consensus_queue.cc:260] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.658533 11910 raft_consensus.cc:729] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d40cd73dd1843c6bf99220b6d4e5583, State: Initialized, Role: FOLLOWER
I20260927 00:33:40.658643 11910 consensus_queue.cc:260] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.658759 11908 ts_tablet_manager.cc:1428] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.658900 11910 ts_tablet_manager.cc:1428] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:40.659199 11916 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.659333 11916 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.658763 11914 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.659559 11914 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.772537 11916 raft_consensus.cc:482] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:40.772699 11916 raft_consensus.cc:504] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.773183 11916 leader_election.cc:288] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457), a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177)
I20260927 00:33:40.773530 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51afda3f8c0443f59d100a99c378f131" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3756bdd2e20a408aaeee5c735f87409f" is_pre_election: true
I20260927 00:33:40.773595 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51afda3f8c0443f59d100a99c378f131" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10" is_pre_election: true
I20260927 00:33:40.773669 11699 raft_consensus.cc:2454] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 0.
I20260927 00:33:40.773741 11839 raft_consensus.cc:2454] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 0.
I20260927 00:33:40.773939 11490 leader_election.cc:302] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, 3756bdd2e20a408aaeee5c735f87409f; no voters:
I20260927 00:33:40.774115 11916 raft_consensus.cc:2781] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:40.774178 11916 raft_consensus.cc:482] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:40.774246 11916 raft_consensus.cc:3037] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.775285 11916 raft_consensus.cc:504] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.775616 11916 leader_election.cc:288] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 election: Requested vote from peers 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457), a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177)
I20260927 00:33:40.775870 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51afda3f8c0443f59d100a99c378f131" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3756bdd2e20a408aaeee5c735f87409f"
I20260927 00:33:40.775943 11839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51afda3f8c0443f59d100a99c378f131" candidate_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18711cb23c5434c9008022a888cbc10"
I20260927 00:33:40.776001 11699 raft_consensus.cc:3037] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.776057 11839 raft_consensus.cc:3037] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:40.776852 11699 raft_consensus.cc:2454] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 1.
I20260927 00:33:40.777118 11839 raft_consensus.cc:2454] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d40cd73dd1843c6bf99220b6d4e5583 in term 1.
I20260927 00:33:40.777133 11490 leader_election.cc:302] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d40cd73dd1843c6bf99220b6d4e5583, 3756bdd2e20a408aaeee5c735f87409f; no voters:
I20260927 00:33:40.777302 11916 raft_consensus.cc:2781] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:40.777398 11916 raft_consensus.cc:686] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 LEADER]: Becoming Leader. State: Replica: 2d40cd73dd1843c6bf99220b6d4e5583, State: Running, Role: LEADER
I20260927 00:33:40.777524 11916 consensus_queue.cc:237] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } }
I20260927 00:33:40.778201 11940 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: TxnStatusTablet state changed. Reason: New leader 2d40cd73dd1843c6bf99220b6d4e5583. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.778311 11940 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:40.778569 11953 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:33:40.778694 11953 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:33:40.779132 11399 catalog_manager.cc:5495] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } health_report { overall_health: UNKNOWN } } }
I20260927 00:33:40.810725 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.807049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.10.11.193:44437)
0927 00:33:40.807152 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.807155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.807167 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:33:40.807330 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.807332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.807379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.807467 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:40.807703 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.808169 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.809342 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.809575 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.809657 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.809971 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.810401 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.810485 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.810589 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.810633 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:33:40.810781 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.806920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44437 (local address 127.0.0.1:33452)
0927 00:33:40.807079 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.807094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.807191 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.807493 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.807496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.807503 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:40.807606 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.807611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.808267 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.808270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.809204 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.809214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.809679 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.809683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.809778 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.809826 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.810486 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.810500 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.810572 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.810648 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:33:40.813220 11839 raft_consensus.cc:1261] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Refusing update from remote peer 2d40cd73dd1843c6bf99220b6d4e5583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.813330 11699 raft_consensus.cc:1261] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Refusing update from remote peer 2d40cd73dd1843c6bf99220b6d4e5583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.813680 11940 consensus_queue.cc:1035] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.813834 11916 consensus_queue.cc:1035] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.816557 11940 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.816656 11940 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:40.817747 11927 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: TxnStatusTablet state changed. Reason: New leader 2d40cd73dd1843c6bf99220b6d4e5583. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.817860 11927 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.818274 11914 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: TxnStatusTablet state changed. Reason: New leader 2d40cd73dd1843c6bf99220b6d4e5583. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.818380 11914 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.818941 11940 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.819029 11940 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 2d40cd73dd1843c6bf99220b6d4e5583: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:40.819739 11927 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.819832 11927 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P 3756bdd2e20a408aaeee5c735f87409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.820014 11914 tablet_replica.cc:439] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d40cd73dd1843c6bf99220b6d4e5583" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44437 } } peers { permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 } } peers { permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 } } }
I20260927 00:33:40.820082 11924 mvcc.cc:205] Tried to move back new op lower bound from 7333761928446627840 to 7333761928306016256. Current Snapshot: MvccSnapshot[applied={T|T < 7333761928446627840}]
I20260927 00:33:40.820101 11914 tablet_replica.cc:442] T 51afda3f8c0443f59d100a99c378f131 P a18711cb23c5434c9008022a888cbc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:40.826695 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.823820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.10.11.193:44437)
0927 00:33:40.823936 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.823943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.823957 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:33:40.823993 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.823996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.824038 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.824112 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:40.824330 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.824818 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.825595 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.825913 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.826021 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.826296 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.826381 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.826455 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.826541 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.826582 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:33:40.836187 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.832659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:53745 (local address 127.10.11.193:44437)
0927 00:33:40.832756 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.832763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.832983 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:33:40.833100 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.833102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.833135 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.833193 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:40.833470 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.833967 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.835171 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.835379 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.835466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.835790 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.835847 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.835905 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.836016 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.836096 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":34}
I20260927 00:33:40.836287 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.832590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44437 (local address 127.10.11.193:53745)
0927 00:33:40.832909 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:40.832922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:40.833023 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:40.833223 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:40.833226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:40.833246 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:40.833374 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.833380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.834057 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.834060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.835048 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:40.835058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.835501 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.835504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:40.835594 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.835637 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:40.835925 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:40.835931 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:33:40.836001 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:33:40.836101 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":6,"thread_start_us":97,"threads_started":1}
I20260927 00:33:40.842126 11699 raft_consensus.cc:1261] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Refusing update from remote peer 2d40cd73dd1843c6bf99220b6d4e5583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.842105 11839 raft_consensus.cc:1261] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Refusing update from remote peer 2d40cd73dd1843c6bf99220b6d4e5583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:40.842470 11940 consensus_queue.cc:1035] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3756bdd2e20a408aaeee5c735f87409f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.842581 11916 consensus_queue.cc:1035] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18711cb23c5434c9008022a888cbc10" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:40.879592 11399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:40.879741 11399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:40.881747 11399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0800c28bdf60471cb2336aefd56b7288: Sending DeleteTablet for 3 replicas of tablet d97f99270a5545b394b8cb012ed682f0
I20260927 00:33:40.882319 11679 tablet_service.cc:1514] Processing DeleteTablet for tablet d97f99270a5545b394b8cb012ed682f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:39588
I20260927 00:33:40.882475 11934 tablet_replica.cc:330] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: stopping tablet replica
I20260927 00:33:40.882498 11819 tablet_service.cc:1514] Processing DeleteTablet for tablet d97f99270a5545b394b8cb012ed682f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:50644
I20260927 00:33:40.882568 11934 raft_consensus.cc:2229] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:40.882660 11934 raft_consensus.cc:2258] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.882684 11932 tablet_replica.cc:330] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: stopping tablet replica
I20260927 00:33:40.882807 11932 raft_consensus.cc:2229] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:40.882925 11932 raft_consensus.cc:2258] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.882898 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet d97f99270a5545b394b8cb012ed682f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:40 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:33448
I20260927 00:33:40.883070 11933 tablet_replica.cc:330] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: stopping tablet replica
I20260927 00:33:40.883189 11933 raft_consensus.cc:2229] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:40.883448 11933 raft_consensus.cc:2258] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:40.883692 11932 ts_tablet_manager.cc:1905] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.884064 11934 ts_tablet_manager.cc:1905] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.884353 11933 ts_tablet_manager.cc:1905] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:40.886830 11933 ts_tablet_manager.cc:1918] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:40.886857 11934 ts_tablet_manager.cc:1918] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:40.886911 11933 log.cc:1196] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/d97f99270a5545b394b8cb012ed682f0
I20260927 00:33:40.886937 11934 log.cc:1196] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/d97f99270a5545b394b8cb012ed682f0
I20260927 00:33:40.887171 11933 ts_tablet_manager.cc:1939] T d97f99270a5545b394b8cb012ed682f0 P 2d40cd73dd1843c6bf99220b6d4e5583: Deleting consensus metadata
I20260927 00:33:40.887244 11934 ts_tablet_manager.cc:1939] T d97f99270a5545b394b8cb012ed682f0 P 3756bdd2e20a408aaeee5c735f87409f: Deleting consensus metadata
I20260927 00:33:40.887359 11932 ts_tablet_manager.cc:1918] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:40.887497 11932 log.cc:1196] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/d97f99270a5545b394b8cb012ed682f0
I20260927 00:33:40.887780 11388 catalog_manager.cc:4841] TS 2d40cd73dd1843c6bf99220b6d4e5583 (127.10.11.193:44437): tablet d97f99270a5545b394b8cb012ed682f0 (table test-table [id=20f39b0eddda40b29a571109b4a351e4]) successfully deleted
I20260927 00:33:40.887950 11386 catalog_manager.cc:4841] TS 3756bdd2e20a408aaeee5c735f87409f (127.10.11.194:42457): tablet d97f99270a5545b394b8cb012ed682f0 (table test-table [id=20f39b0eddda40b29a571109b4a351e4]) successfully deleted
I20260927 00:33:40.888190 11932 ts_tablet_manager.cc:1939] T d97f99270a5545b394b8cb012ed682f0 P a18711cb23c5434c9008022a888cbc10: Deleting consensus metadata
I20260927 00:33:40.888827 11387 catalog_manager.cc:4841] TS a18711cb23c5434c9008022a888cbc10 (127.10.11.195:43177): tablet d97f99270a5545b394b8cb012ed682f0 (table test-table [id=20f39b0eddda40b29a571109b4a351e4]) successfully deleted
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:40.899048 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.883734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.10.11.193:44437)
0927 00:33:40.883830 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.883835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.888987 (+ 5152us) server_negotiation.cc:407] Connection header received
0927 00:33:40.889128 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.889131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.889177 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.889272 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:40.889981 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.890771 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.891610 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.891957 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.892058 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.895297 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:40.895306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:40.895321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:40.895349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:40.897211 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.897632 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.897638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.897640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.897740 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.897995 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.897998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.898000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.898297 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:40.898523 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.898725 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.898839 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 00:33:40.899510 11586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33470
I20260927 00:33:40.908890 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:40.901137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.10.11.254:44249)
0927 00:33:40.901272 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:40.901277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:40.901470 (+ 193us) server_negotiation.cc:407] Connection header received
0927 00:33:40.901548 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:40.901551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:40.901597 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:40.901688 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:40.902340 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.903082 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.903939 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:40.904269 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:40.904378 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:40.905648 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:40.905653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:40.905658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:40.905690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:40.907234 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.907676 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.907680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.907682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.907773 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:40.908058 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:40.908063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:40.908065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:40.908285 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:40.908418 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:40.908593 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:33:40.908714 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:33:40.909198 11399 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51170
I20260927 00:33:40.910470 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11474
I20260927 00:33:40.919212 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11614
I20260927 00:33:40.925282 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11754
I20260927 00:33:40.930866 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11370
2026-09-27T00:33:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2977 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11971](info): set up 2 sockets
Sep 27 00:33:40 dist-test-slave-6gxc krb5kdc[11971](info): commencing operation
krb5kdc: starting...
W20260927 00:33:41.974421 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:42.088599 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:44193
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:42.195626 11984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:42.195861 11984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:42.195900 11984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:42.199761 11984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:42.199810 11984 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:33:42.199831 11984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:42.199851 11984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:42.199869 11984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:42.199887 11984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:42.204836 11984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:44193
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:42.205637 11984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:42.206553 11984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:42.214275 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.214286 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.214274 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.214582 11984 server_base.cc:1034] running on GCE node
I20260927 00:33:42.215129 11984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:42.216095 11984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:42.217247 11984 hybrid_clock.cc:648] HybridClock initialized: now 1790469222217227 us; error 32 us; skew 500 ppm
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:42.224717 11984 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:42.225572 11984 webserver.cc:460] Webserver started at http://127.10.11.254:46661/ using document root <none> and password file <none>
I20260927 00:33:42.226089 11984 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:42.226224 11984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:42.226471 11984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:42.228370 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "000e832767ce4e92b39cc4099fca1da4"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "15c84e0204833b626e04c733ab7a9af9"
server_key_iv: "30dfbba57ad657679d32c1b8f9082ed0"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.228936 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "000e832767ce4e92b39cc4099fca1da4"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "15c84e0204833b626e04c733ab7a9af9"
server_key_iv: "30dfbba57ad657679d32c1b8f9082ed0"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.232493 11984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:33:42.234790 11999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.235908 11984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:33:42.236063 11984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "000e832767ce4e92b39cc4099fca1da4"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "15c84e0204833b626e04c733ab7a9af9"
server_key_iv: "30dfbba57ad657679d32c1b8f9082ed0"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.236181 11984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:42.262866 11984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:42.265466 11984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:42.265653 11984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:42.273167 11984 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:44193
I20260927 00:33:42.273182 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:44193 every 8 connection(s)
I20260927 00:33:42.274022 11984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:42.276566 12062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:42.281524 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Bootstrap starting.
I20260927 00:33:42.283732 12062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:42.284263 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 11984
I20260927 00:33:42.284374 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:42.284588 12062 log.cc:824] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:42.284633 10287 external_mini_cluster.cc:1428] Setting key 3fe264282ea91148442eed198150b0d3
I20260927 00:33:42.286594 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: No bootstrap required, opened a new log
I20260927 00:33:42.290000 12062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } }
I20260927 00:33:42.290252 12062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:42.290325 12062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 000e832767ce4e92b39cc4099fca1da4, State: Initialized, Role: FOLLOWER
I20260927 00:33:42.290760 12062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } }
I20260927 00:33:42.290915 12062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:42.290977 12062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:42.291071 12062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:42.291988 12062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } }
I20260927 00:33:42.292306 12062 leader_election.cc:302] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 000e832767ce4e92b39cc4099fca1da4; no voters:
I20260927 00:33:42.292622 12062 leader_election.cc:288] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:42.292716 12067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:42.292958 12067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [term 1 LEADER]: Becoming Leader. State: Replica: 000e832767ce4e92b39cc4099fca1da4, State: Running, Role: LEADER
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:42.293246 12067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } }
I20260927 00:33:42.293826 12062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:42.294854 12069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "000e832767ce4e92b39cc4099fca1da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } } }
I20260927 00:33:42.294977 12069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:42.295372 12068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 000e832767ce4e92b39cc4099fca1da4. Latest consensus state: current_term: 1 leader_uuid: "000e832767ce4e92b39cc4099fca1da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000e832767ce4e92b39cc4099fca1da4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 44193 } } }
I20260927 00:33:42.295550 12068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:42.295336 12076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:42.298898 12076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:42.299329 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.285650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.10.11.254:44193)
0927 00:33:42.286047 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.286061 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.286092 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:33:42.286822 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.286838 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.287125 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.287452 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.288233 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.289316 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.290215 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.290791 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.290952 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.293706 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.293712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.293727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.293756 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.295829 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.296347 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.296354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.296359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.296507 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.297241 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.297244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.297246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.297685 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.297872 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.298187 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.298464 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":147,"threads_started":1}
I20260927 00:33:42.303681 12076 catalog_manager.cc:1347] Generated new cluster ID: 714ee6613b8f48b79489c30f369e5bed
I20260927 00:33:42.303740 12076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:42.321525 12076 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:42.322736 12076 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:42.334409 12076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Generated new TSK 0
I20260927 00:33:42.335106 12076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:42.365013 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:42.471211 12088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:42.471437 12088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:42.471522 12088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:42.475494 12088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:42.475548 12088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:42.475590 12088 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:33:42.475646 12088 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:42.475720 12088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:42.480715 12088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:42.481576 12088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:42.482513 12088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:42.491228 12096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.491228 12094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.491385 12088 server_base.cc:1034] running on GCE node
W20260927 00:33:42.491290 12093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.491776 12088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:42.492316 12088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:42.493492 12088 hybrid_clock.cc:648] HybridClock initialized: now 1790469222493474 us; error 29 us; skew 500 ppm
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:42.496189 12088 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:42.496930 12088 webserver.cc:460] Webserver started at http://127.10.11.193:40811/ using document root <none> and password file <none>
I20260927 00:33:42.497430 12088 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:42.497515 12088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:42.497735 12088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:42.499548 12088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "a4e1735e27f24df48c6b498b420865be"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "03fdf1bd7d6d314c7ea7c850e9617260"
server_key_iv: "a60e633b90c1d87a1ece48b8818750b5"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.500124 12088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "a4e1735e27f24df48c6b498b420865be"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "03fdf1bd7d6d314c7ea7c850e9617260"
server_key_iv: "a60e633b90c1d87a1ece48b8818750b5"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.503464 12088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 00:33:42.505609 12103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.506759 12088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:33:42.506892 12088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "a4e1735e27f24df48c6b498b420865be"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "03fdf1bd7d6d314c7ea7c850e9617260"
server_key_iv: "a60e633b90c1d87a1ece48b8818750b5"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.506994 12088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:42.520741 12088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:42.523553 12088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:42.523734 12088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:42.524935 12088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:42.525004 12088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.525068 12088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:42.525120 12088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:42.535352 12088 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:39585
I20260927 00:33:42.535759 12218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:39585 every 8 connection(s)
I20260927 00:33:42.536170 12088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:42.538458 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.525925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:56275 (local address 127.10.11.254:44193)
0927 00:33:42.526076 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.526081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.526945 (+ 864us) server_negotiation.cc:407] Connection header received
0927 00:33:42.528166 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.528169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.528246 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.528347 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.529751 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.530656 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.531921 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.532188 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.532309 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.534840 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.534844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.534846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.534876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.536426 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.536936 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.536942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.536943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.537049 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.537422 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.537426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.537428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.537593 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.537713 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.538176 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.538301 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:33:42.539166 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.526357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.193:56275)
0927 00:33:42.526781 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.526826 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.527900 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.528528 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.528536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.528928 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.529569 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.529616 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.530787 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.530791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.531800 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.531810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.532352 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.532356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.532520 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.533048 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.533063 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.534664 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.536534 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.536540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.536572 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.536836 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.537168 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.537171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.537173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.537316 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.537719 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.537726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.537934 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.538103 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.538391 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
I20260927 00:33:42.540307 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12088
I20260927 00:33:42.540421 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:42.540577 12219 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44193
I20260927 00:33:42.540709 10287 external_mini_cluster.cc:1428] Setting key 29d7db9757471b66548de27ac34b584a
I20260927 00:33:42.540731 12219 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:42.541184 12219 heartbeater.cc:511] Master 127.10.11.254:44193 requested a full tablet report, sending...
I20260927 00:33:42.542743 12016 ts_manager.cc:194] Registered new tserver with Master: a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585)
I20260927 00:33:42.544216 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:56275
I20260927 00:33:42.550204 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.542842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:51263 (local address 127.10.11.254:44193)
0927 00:33:42.542960 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.542966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.543173 (+ 207us) server_negotiation.cc:407] Connection header received
0927 00:33:42.543275 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.543281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.543342 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.543420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.543889 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.544563 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.545718 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.546149 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.546270 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.547198 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.547201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.547203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.547242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.548647 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.549074 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.549077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.549079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.549152 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.549426 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.549429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.549431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.549587 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.549682 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.549938 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.550036 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:33:42.550217 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.542781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.193:51263)
0927 00:33:42.543083 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.543097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.543184 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.543456 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.543459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.543652 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.543765 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.543771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.544733 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.544736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.545592 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.545603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.546260 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.546263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.546350 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.546796 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.546807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.547082 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.548771 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.548775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.548777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.548970 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.549237 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.549241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.549244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.549329 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.549684 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.549688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.549776 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.549887 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.550012 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:42.567232 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:42.670861 12228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:42.671067 12228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:42.671109 12228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:42.675076 12228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:42.675124 12228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:42.675160 12228 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:33:42.675174 12228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:42.675242 12228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:42.680075 12228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:42.680904 12228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:42.681810 12228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:42.690745 12236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.690733 12234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.690814 12233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.690776 12228 server_base.cc:1034] running on GCE node
I20260927 00:33:42.691260 12228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:42.691838 12228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:42.693033 12228 hybrid_clock.cc:648] HybridClock initialized: now 1790469222693021 us; error 24 us; skew 500 ppm
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:42.695776 12228 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:42.696569 12228 webserver.cc:460] Webserver started at http://127.10.11.194:35457/ using document root <none> and password file <none>
I20260927 00:33:42.697080 12228 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:42.697165 12228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:42.697371 12228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:42.699234 12228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "330bb3c3217440189b5de7e10a574f8f"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "41c4fdf3a820ddb7d04724b812aa5d3a"
server_key_iv: "68f516fb6f0065241c3aa6daf393b362"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.699767 12228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "330bb3c3217440189b5de7e10a574f8f"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "41c4fdf3a820ddb7d04724b812aa5d3a"
server_key_iv: "68f516fb6f0065241c3aa6daf393b362"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.703210 12228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:42.705339 12243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.706525 12228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:42.706655 12228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "330bb3c3217440189b5de7e10a574f8f"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "41c4fdf3a820ddb7d04724b812aa5d3a"
server_key_iv: "68f516fb6f0065241c3aa6daf393b362"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.706755 12228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:42.728765 12228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:42.731607 12228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:42.731807 12228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:42.733147 12228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:42.733215 12228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.733289 12228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:42.733338 12228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:42.744196 12228 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:40357
I20260927 00:33:42.744428 12358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:40357 every 8 connection(s)
I20260927 00:33:42.745139 12228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:42.747028 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.733890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:41039 (local address 127.10.11.254:44193)
0927 00:33:42.734040 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.734045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.734974 (+ 929us) server_negotiation.cc:407] Connection header received
0927 00:33:42.735928 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.735931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.735993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.736054 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.737459 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.738021 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.739506 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.739919 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.740040 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.743175 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.743180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.743184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.743213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.744880 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.745439 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.745442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.745443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.745517 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.746020 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.746025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.746030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.746276 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.746358 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.746828 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.746918 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:33:42.747821 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.734362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.194:41039)
0927 00:33:42.734810 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.734858 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.735668 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.736465 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.736471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.736774 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.737305 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.737325 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.738206 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.738210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.739388 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.739399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.740099 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.740105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.740327 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.740958 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.740982 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.742987 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.744984 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.744989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.745002 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.745323 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.745743 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.745746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.745748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.745903 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.746384 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.746388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.746596 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.746779 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.747008 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260927 00:33:42.748970 12359 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44193
I20260927 00:33:42.749105 12359 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:42.749651 12359 heartbeater.cc:511] Master 127.10.11.254:44193 requested a full tablet report, sending...
I20260927 00:33:42.750687 12015 ts_manager.cc:194] Registered new tserver with Master: 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357)
I20260927 00:33:42.751431 12015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:41039
I20260927 00:33:42.752525 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12228
I20260927 00:33:42.752633 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:42.752884 10287 external_mini_cluster.cc:1428] Setting key 6beed7d9820af79dfa6d0e9238807710
I20260927 00:33:42.758730 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.751135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:52735 (local address 127.10.11.254:44193)
0927 00:33:42.751243 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.751246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.751886 (+ 640us) server_negotiation.cc:407] Connection header received
0927 00:33:42.752000 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.752003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.752046 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.752126 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.752626 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.753378 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.754353 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.754561 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.754667 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.755708 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.755712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.755714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.755733 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.757104 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.757526 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.757535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.757536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.757603 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.757876 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.757881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.757883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.758079 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.758233 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.758489 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.758592 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:33:42.758739 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.751073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.194:52735)
0927 00:33:42.751792 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.751806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.751906 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.752145 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.752147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.752351 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.752495 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.752504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.753512 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.753515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.754241 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.754250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.754668 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.754672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.754763 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.755250 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.755269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.755555 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.757221 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.757224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.757226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.757420 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.757698 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.757701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.757703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.757790 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.758224 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.758227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.758324 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.758426 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.758558 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:42.777359 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:44193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:42.883565 12368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:42.883787 12368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:42.883867 12368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:42.887733 12368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:42.887782 12368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:42.887820 12368 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:33:42.887872 12368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:33:42.887928 12368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:42.892927 12368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:42.893797 12368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:42.894706 12368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:42.903513 12376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:42.903507 12373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.903573 12368 server_base.cc:1034] running on GCE node
W20260927 00:33:42.903507 12374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:42.904114 12368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:42.904670 12368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:42.905833 12368 hybrid_clock.cc:648] HybridClock initialized: now 1790469222905807 us; error 41 us; skew 500 ppm
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:42.908468 12368 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:42.909368 12368 webserver.cc:460] Webserver started at http://127.10.11.195:45835/ using document root <none> and password file <none>
I20260927 00:33:42.909971 12368 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:42.910053 12368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:42.910297 12368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:42.912217 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "427ae3736ef64b7a8bb4f31a1fb07a82"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "7f1454e972494aad75ba61a9b948b897"
server_key_iv: "eb711d6bc44113aaa577bec3e4afa23c"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.912736 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "427ae3736ef64b7a8bb4f31a1fb07a82"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "7f1454e972494aad75ba61a9b948b897"
server_key_iv: "eb711d6bc44113aaa577bec3e4afa23c"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.916137 12368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:33:42.918383 12383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.919471 12368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:33:42.919592 12368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "427ae3736ef64b7a8bb4f31a1fb07a82"
format_stamp: "Formatted at 2026-09-27 00:33:42 on dist-test-slave-6gxc"
server_key: "7f1454e972494aad75ba61a9b948b897"
server_key_iv: "eb711d6bc44113aaa577bec3e4afa23c"
server_key_version: "encryptionkey@0"
I20260927 00:33:42.919711 12368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:42.933815 12368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:42.936740 12368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:42.936955 12368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:42.938271 12368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:42.938345 12368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:42.938414 12368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:42.938449 12368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:42.948741 12368 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:46099
I20260927 00:33:42.948774 12498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:46099 every 8 connection(s)
I20260927 00:33:42.949647 12368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:42.952409 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.939143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:39309 (local address 127.10.11.254:44193)
0927 00:33:42.939317 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.939323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.940193 (+ 870us) server_negotiation.cc:407] Connection header received
0927 00:33:42.941221 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.941225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.941290 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.941384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.942973 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.943837 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.945677 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.946012 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.946145 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.949006 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.949011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.949016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.949045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.950690 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.951116 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.951119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.951120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.951194 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.951486 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.951488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.951489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.951644 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.951715 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.952131 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.952219 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:33:42.952729 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12368
I20260927 00:33:42.952812 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:42.953075 10287 external_mini_cluster.cc:1428] Setting key 553e7ec3586360875f904b83936292bd
I20260927 00:33:42.953902 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.939564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.195:39309)
0927 00:33:42.940035 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.940084 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.940982 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.941656 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.941667 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.942155 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.942817 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.942839 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.943974 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.943980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.945523 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.945536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.946192 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.946195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.946417 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.947056 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.947082 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.948817 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.950786 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.950789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.950799 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.951028 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.951288 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.951290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.951291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.951407 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.951734 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.951738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.951939 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.952085 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.952312 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260927 00:33:42.956656 12499 heartbeater.cc:347] Connected to a master server at 127.10.11.254:44193
I20260927 00:33:42.956820 12499 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:42.957345 12499 heartbeater.cc:511] Master 127.10.11.254:44193 requested a full tablet report, sending...
I20260927 00:33:42.958251 12015 ts_manager.cc:194] Registered new tserver with Master: 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099)
I20260927 00:33:42.958832 12015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:39309
I20260927 00:33:42.964191 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.956940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:50645 (local address 127.10.11.254:44193)
0927 00:33:42.957053 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.957059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.957606 (+ 547us) server_negotiation.cc:407] Connection header received
0927 00:33:42.957716 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.957719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.957763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.957847 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.958356 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.958863 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.959742 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.959957 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.960062 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.961055 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.961058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.961061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.961083 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.962457 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.962940 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.962947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.962949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.963033 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.963335 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.963341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.963343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.963515 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.963625 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.963914 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.964018 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:33:42.964191 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.956916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.10.11.195:50645)
0927 00:33:42.957519 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:42.957539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:42.957642 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:42.957862 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:42.957865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:42.958078 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:42.958219 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.958226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.959025 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.959030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.959626 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:42.959633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.960065 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.960069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:42.960173 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.960602 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:42.960612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:42.960929 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:42.962573 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.962579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.962582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.962833 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.963134 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:42.963138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:42.963141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:42.963234 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:42.963626 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:42.963629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:42.963737 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:33:42.963846 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:33:42.963983 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":206,"threads_started":1}
I20260927 00:33:42.966849 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:33:42 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:42.990813 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:42.982996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.10.11.254:44193)
0927 00:33:42.983116 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:42.983119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:42.983132 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:33:42.983166 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:42.983168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:42.983207 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:42.983280 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:42.983896 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.984377 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.985183 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:42.985390 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:42.985470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:42.987765 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:42.987767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:42.987769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:42.987814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:42.989223 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.989578 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.989580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.989581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.989647 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:42.989879 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:42.989881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:42.989882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:42.990174 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:42.990271 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:42.990510 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:33:42.990591 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:33:42.993469 12015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:42.996553 12015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:43.010075 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.004333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40357 (local address 127.0.0.1:46106)
0927 00:33:43.004898 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.004911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.004986 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.006366 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.006369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.006383 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.006473 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.006479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007543 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.007546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.008545 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.008554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.009612 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.009615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.009724 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.009761 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.009840 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.009939 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":65,"thread_start_us":84,"threads_started":1}
I20260927 00:33:43.010426 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.004240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:46099 (local address 127.0.0.1:38018)
0927 00:33:43.004532 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.004574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.004898 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.005929 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.005936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.005959 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.006075 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.006081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007469 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.007475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.008946 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.008956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.010005 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.010008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.010138 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.010159 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.010225 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.010293 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":42,"thread_start_us":88,"threads_started":1}
I20260927 00:33:43.010491 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.004337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.10.11.195:46099)
0927 00:33:43.005435 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.005448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.005472 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:33:43.005545 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.005551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.005747 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.005914 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.006197 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007342 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.009073 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.009891 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.010004 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.010229 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.010277 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.010362 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":85,"threads_started":1}
I20260927 00:33:43.010807 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.004925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39585 (local address 127.0.0.1:37110)
0927 00:33:43.005147 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.005160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.005233 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.006254 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.006258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.006279 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.006378 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.006384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007791 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.007794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.008994 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.009003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.010301 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.010305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.010403 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.010426 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.010477 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.010661 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260927 00:33:43.011335 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.005136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.10.11.194:40357)
0927 00:33:43.005888 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.005900 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.005920 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:33:43.005985 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.005990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.006214 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.006596 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.006700 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007414 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.008674 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.009477 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.010846 (+ 1369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.011021 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.011079 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.011164 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":84,"threads_started":1}
I20260927 00:33:43.011526 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.005519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.10.11.193:39585)
0927 00:33:43.005783 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.005795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.005820 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:33:43.005891 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.005896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.006070 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.006297 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.006467 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.007543 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.009306 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.010176 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.011062 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.011190 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.011292 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.011386 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260927 00:33:43.013422 12433 tablet_service.cc:1467] Processing CreateTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda (DEFAULT_TABLE table=test-table [id=30f65ff5ce7e4449ab43771387f7672c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.013701 12153 tablet_service.cc:1467] Processing CreateTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda (DEFAULT_TABLE table=test-table [id=30f65ff5ce7e4449ab43771387f7672c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.013908 12293 tablet_service.cc:1467] Processing CreateTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda (DEFAULT_TABLE table=test-table [id=30f65ff5ce7e4449ab43771387f7672c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.014519 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff2bdcdf0b24e309dcc24d40c2d2fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.014668 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff2bdcdf0b24e309dcc24d40c2d2fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.014804 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff2bdcdf0b24e309dcc24d40c2d2fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.020655 12522 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Bootstrap starting.
I20260927 00:33:43.022522 12523 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Bootstrap starting.
I20260927 00:33:43.023042 12522 tablet_bootstrap.cc:654] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.023819 12522 log.cc:824] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:43.024389 12524 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Bootstrap starting.
I20260927 00:33:43.024407 12523 tablet_bootstrap.cc:654] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.025028 12523 log.cc:824] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:43.026095 12524 tablet_bootstrap.cc:654] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.026856 12523 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: No bootstrap required, opened a new log
I20260927 00:33:43.027004 12524 log.cc:824] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:43.027037 12523 ts_tablet_manager.cc:1397] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:33:43.027302 12522 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: No bootstrap required, opened a new log
I20260927 00:33:43.027530 12522 ts_tablet_manager.cc:1397] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:33:43.028527 12524 tablet_bootstrap.cc:492] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: No bootstrap required, opened a new log
I20260927 00:33:43.028713 12524 ts_tablet_manager.cc:1397] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 00:33:43.030875 12523 raft_consensus.cc:348] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.031124 12523 raft_consensus.cc:374] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.031212 12523 raft_consensus.cc:729] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.031258 12522 raft_consensus.cc:348] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.031527 12522 raft_consensus.cc:374] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.031584 12522 raft_consensus.cc:729] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1735e27f24df48c6b498b420865be, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.031656 12523 consensus_queue.cc:260] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.031562 12524 raft_consensus.cc:348] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.031785 12524 raft_consensus.cc:374] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.031843 12524 raft_consensus.cc:729] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.031968 12522 consensus_queue.cc:260] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.032248 12524 consensus_queue.cc:260] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.032586 12523 ts_tablet_manager.cc:1428] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:33:43.032778 12522 ts_tablet_manager.cc:1428] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:33:43.032816 12219 heartbeater.cc:503] Master 127.10.11.254:44193 was elected leader, sending a full tablet report...
I20260927 00:33:43.032941 12359 heartbeater.cc:503] Master 127.10.11.254:44193 was elected leader, sending a full tablet report...
I20260927 00:33:43.033162 12499 heartbeater.cc:503] Master 127.10.11.254:44193 was elected leader, sending a full tablet report...
I20260927 00:33:43.033162 12524 ts_tablet_manager.cc:1428] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260927 00:33:43.037710 12220 tablet.cc:2369] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:43.142573 12530 raft_consensus.cc:482] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:43.142776 12530 raft_consensus.cc:504] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.143970 12530 leader_election.cc:288] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357), a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585)
I20260927 00:33:43.147396 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.144152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40357 (local address 127.10.11.195:46853)
0927 00:33:43.144275 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.144289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.144377 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.144636 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.144639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.144655 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.144770 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.144776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.145508 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.145513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.146334 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.146343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.146985 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.146988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.147096 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.147113 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.147199 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.147270 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:33:43.147396 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.144255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:46853 (local address 127.10.11.194:40357)
0927 00:33:43.144390 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.144394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.144411 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:33:43.144463 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.144466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.144527 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.144623 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.144895 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.145395 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.146461 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.146854 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.146984 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.147037 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.147214 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.147279 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:33:43.148094 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.144434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:37991 (local address 127.10.11.193:39585)
0927 00:33:43.144552 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.144556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.144711 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:33:43.144855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.144858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.144915 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.144997 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.145594 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.146076 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.147082 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.147520 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.147653 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.147704 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.147948 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.148008 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:33:43.148103 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.144219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39585 (local address 127.10.11.195:37991)
0927 00:33:43.144620 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.144631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.144740 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.145008 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.145011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.145026 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.145469 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.145476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.146210 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.146216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.146958 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.146966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.147688 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.147692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.147794 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.147817 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.147919 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.147981 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":131,"threads_started":1}
I20260927 00:33:43.148132 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff2bdcdf0b24e309dcc24d40c2d2fda" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f" is_pre_election: true
I20260927 00:33:43.148419 12313 raft_consensus.cc:2454] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:43.148800 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff2bdcdf0b24e309dcc24d40c2d2fda" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be" is_pre_election: true
I20260927 00:33:43.148923 12387 leader_election.cc:302] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82; no voters:
I20260927 00:33:43.149094 12173 raft_consensus.cc:2454] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:43.149233 12530 raft_consensus.cc:2781] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:43.149325 12530 raft_consensus.cc:482] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:43.149401 12530 raft_consensus.cc:3037] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.150523 12530 raft_consensus.cc:504] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.150835 12530 leader_election.cc:288] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Requested vote from peers 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357), a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585)
I20260927 00:33:43.151100 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff2bdcdf0b24e309dcc24d40c2d2fda" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f"
I20260927 00:33:43.151232 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff2bdcdf0b24e309dcc24d40c2d2fda" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be"
I20260927 00:33:43.151238 12313 raft_consensus.cc:3037] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.151341 12173 raft_consensus.cc:3037] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.152526 12173 raft_consensus.cc:2454] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:43.152526 12313 raft_consensus.cc:2454] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:43.152807 12387 leader_election.cc:302] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82; no voters:
I20260927 00:33:43.153014 12530 raft_consensus.cc:2781] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:43.153308 12530 raft_consensus.cc:686] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Becoming Leader. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Running, Role: LEADER
I20260927 00:33:43.153636 12530 consensus_queue.cc:237] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } }
I20260927 00:33:43.156553 12016 catalog_manager.cc:5495] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195). New cstate: current_term: 1 leader_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } health_report { overall_health: UNKNOWN } } }
W20260927 00:33:43.201191 12500 tablet.cc:2369] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:33:43 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469222, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:43.239411 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.231594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:44193 (local address 127.0.0.1:44988)
0927 00:33:43.231873 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.231885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.231977 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.232198 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.232200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.232582 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:43.232668 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.232674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.233391 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.233393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.233951 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.233959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.234395 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.234398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.234455 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.235035 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:43.235045 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:43.236381 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:43.237889 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:43.237896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:43.237901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:43.238159 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:43.238394 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:43.238396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:43.238397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:43.238470 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:43.238823 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:43.238828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:43.238997 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.239131 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.239256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260927 00:33:43.239430 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.231732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.10.11.254:44193)
0927 00:33:43.231886 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.231889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.231951 (+ 62us) server_negotiation.cc:407] Connection header received
0927 00:33:43.232048 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.232050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.232086 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.232151 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:43.232772 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.233273 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.234048 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.234310 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.234383 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.236510 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:43.236513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:43.236515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:43.236534 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:43.237793 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:43.238244 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:43.238247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:43.238248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:43.238317 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:43.238542 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:43.238544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:43.238545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:43.238708 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:43.238783 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.239183 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.239297 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":56}
I20260927 00:33:43.243042 12016 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:44988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:33:43.243659 12016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 00:33:43.246632 12360 tablet.cc:2369] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:43.248169 12153 tablet_service.cc:1467] Processing CreateTablet for tablet 97ef4dbe5fda4f638ea47053d213db37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d9c86ccac846b3823d26be19f5ebb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:43.248368 12433 tablet_service.cc:1467] Processing CreateTablet for tablet 97ef4dbe5fda4f638ea47053d213db37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d9c86ccac846b3823d26be19f5ebb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:43.248389 12293 tablet_service.cc:1467] Processing CreateTablet for tablet 97ef4dbe5fda4f638ea47053d213db37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d9c86ccac846b3823d26be19f5ebb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:33:43.248929 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ef4dbe5fda4f638ea47053d213db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.248924 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ef4dbe5fda4f638ea47053d213db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.248957 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ef4dbe5fda4f638ea47053d213db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.251406 12523 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: Bootstrap starting.
I20260927 00:33:43.251683 12524 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: Bootstrap starting.
I20260927 00:33:43.251992 12522 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: Bootstrap starting.
I20260927 00:33:43.252446 12523 tablet_bootstrap.cc:654] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.252681 12524 tablet_bootstrap.cc:654] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.252775 12522 tablet_bootstrap.cc:654] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.253643 12523 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: No bootstrap required, opened a new log
I20260927 00:33:43.253726 12523 ts_tablet_manager.cc:1397] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:43.254204 12522 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: No bootstrap required, opened a new log
I20260927 00:33:43.254164 12523 raft_consensus.cc:348] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.254256 12523 raft_consensus.cc:374] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.254261 12522 ts_tablet_manager.cc:1397] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:43.254278 12523 raft_consensus.cc:729] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.254405 12523 consensus_queue.cc:260] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.254563 12522 raft_consensus.cc:348] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.254667 12523 ts_tablet_manager.cc:1428] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:33:43.254673 12522 raft_consensus.cc:374] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.254705 12522 raft_consensus.cc:729] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1735e27f24df48c6b498b420865be, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.254632 12528 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.254765 12528 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.254819 12522 consensus_queue.cc:260] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.255085 12522 ts_tablet_manager.cc:1428] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:43.255350 12529 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.255487 12529 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.255628 12524 tablet_bootstrap.cc:492] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: No bootstrap required, opened a new log
I20260927 00:33:43.255726 12524 ts_tablet_manager.cc:1397] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 00:33:43.256217 12524 raft_consensus.cc:348] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.256327 12524 raft_consensus.cc:374] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.256362 12524 raft_consensus.cc:729] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.256498 12524 consensus_queue.cc:260] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.256771 12524 ts_tablet_manager.cc:1428] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:33:43.256742 12530 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.256861 12530 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.394053 12528 raft_consensus.cc:482] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:43.394243 12528 raft_consensus.cc:504] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.395411 12528 leader_election.cc:288] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099)
I20260927 00:33:43.399407 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.395636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39585 (local address 127.10.11.194:50749)
0927 00:33:43.396113 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.396125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.396202 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.396550 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.396553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.396571 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.396688 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.396697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.397496 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.397501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.398348 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.398356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.398994 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.398999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.399090 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.399112 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.399183 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.399242 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":111,"thread_start_us":44,"threads_started":1}
I20260927 00:33:43.399426 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.395861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:50749 (local address 127.10.11.193:39585)
0927 00:33:43.395976 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.395981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.396194 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:33:43.396293 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.396295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.396339 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.396431 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.396803 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.397356 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.398462 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.398873 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.398967 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.399029 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.399213 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.399296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:33:43.399489 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.395636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:46099 (local address 127.10.11.194:40637)
0927 00:33:43.396022 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.396037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.396152 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.396462 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.396464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.396489 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:43.396618 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.396624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.397451 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.397455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.398393 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.398420 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.399085 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.399088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.399178 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.399204 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.399277 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.399338 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":70,"thread_start_us":115,"threads_started":1}
I20260927 00:33:43.399493 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.395813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40637 (local address 127.10.11.195:46099)
0927 00:33:43.395952 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.395956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.396113 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:33:43.396269 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.396272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.396323 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.396431 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:43.396730 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.397343 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.398530 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.398953 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.399066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.399113 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.399303 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.399371 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:33:43.399828 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ef4dbe5fda4f638ea47053d213db37" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be" is_pre_election: true
I20260927 00:33:43.399968 12173 raft_consensus.cc:2454] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 0.
I20260927 00:33:43.400187 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ef4dbe5fda4f638ea47053d213db37" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" is_pre_election: true
I20260927 00:33:43.400311 12245 leader_election.cc:302] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:43.400396 12439 raft_consensus.cc:2454] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 0.
I20260927 00:33:43.400565 12528 raft_consensus.cc:2781] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:43.400643 12528 raft_consensus.cc:482] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:43.400695 12528 raft_consensus.cc:3037] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.401677 12528 raft_consensus.cc:504] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.402004 12528 leader_election.cc:288] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 election: Requested vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099)
I20260927 00:33:43.402333 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ef4dbe5fda4f638ea47053d213db37" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be"
I20260927 00:33:43.402397 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ef4dbe5fda4f638ea47053d213db37" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82"
I20260927 00:33:43.402458 12173 raft_consensus.cc:3037] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.402487 12439 raft_consensus.cc:3037] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.403582 12439 raft_consensus.cc:2454] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 1.
I20260927 00:33:43.403582 12173 raft_consensus.cc:2454] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 1.
I20260927 00:33:43.403859 12245 leader_election.cc:302] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:43.404008 12528 raft_consensus.cc:2781] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:43.404197 12528 raft_consensus.cc:686] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [term 1 LEADER]: Becoming Leader. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Running, Role: LEADER
I20260927 00:33:43.404520 12528 consensus_queue.cc:237] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.406646 12545 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: TxnStatusTablet state changed. Reason: New leader 330bb3c3217440189b5de7e10a574f8f. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.406747 12545 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:43.407125 12547 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:33:43.407351 12547 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:33:43.407829 12016 catalog_manager.cc:5495] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 330bb3c3217440189b5de7e10a574f8f (127.10.11.194). New cstate: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260927 00:33:43.447079 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.443278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.10.11.194:40357)
0927 00:33:43.443399 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.443403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.443491 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:33:43.443730 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.443733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.443785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.443869 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:43.444131 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.444596 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.445762 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.445963 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.446034 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.446369 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.446733 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.446816 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.446942 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.446982 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:33:43.447119 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.443180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40357 (local address 127.0.0.1:46108)
0927 00:33:43.443406 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.443421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.443526 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.443912 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.443915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.443922 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:43.444023 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.444030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.444697 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.444700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.445650 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.445660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.446066 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.446070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.446180 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.446229 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.446824 (+ 595us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.446837 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.446915 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.446988 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 00:33:43.452059 12439 raft_consensus.cc:1261] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Refusing update from remote peer 330bb3c3217440189b5de7e10a574f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.452108 12173 raft_consensus.cc:1261] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Refusing update from remote peer 330bb3c3217440189b5de7e10a574f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.452654 12545 consensus_queue.cc:1035] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:43.452836 12528 consensus_queue.cc:1035] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:43.455888 12530 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: TxnStatusTablet state changed. Reason: New leader 330bb3c3217440189b5de7e10a574f8f. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.455994 12530 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.457618 12528 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.457726 12528 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:43.459156 12530 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.459280 12530 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 427ae3736ef64b7a8bb4f31a1fb07a82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.461418 12529 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: TxnStatusTablet state changed. Reason: New leader 330bb3c3217440189b5de7e10a574f8f. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.461534 12529 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.463474 12551 mvcc.cc:205] Tried to move back new op lower bound from 7333761939253280768 to 7333761939067592704. Current Snapshot: MvccSnapshot[applied={T|T < 7333761939253280768 or (T in {7333761939253280768})}]
I20260927 00:33:43.465437 12545 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.465533 12545 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P 330bb3c3217440189b5de7e10a574f8f: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:33:43.467294 12529 tablet_replica.cc:439] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } } }
I20260927 00:33:43.467391 12529 tablet_replica.cc:442] T 97ef4dbe5fda4f638ea47053d213db37 P a4e1735e27f24df48c6b498b420865be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:33:43.467746 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.462978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.10.11.195:46099)
0927 00:33:43.463080 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.463084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.463096 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:33:43.463150 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.463152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.463211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.463314 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:43.463576 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.464338 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.465682 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.466028 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.466137 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.466475 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.466941 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.467037 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.467580 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.467628 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:33:43.467857 12555 mvcc.cc:205] Tried to move back new op lower bound from 7333761939253280768 to 7333761939067592704. Current Snapshot: MvccSnapshot[applied={T|T < 7333761939253280768}]
I20260927 00:33:43.477823 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.474340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:59327 (local address 127.10.11.194:40357)
0927 00:33:43.474444 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.474448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.474629 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:33:43.474806 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.474809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.474852 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.474913 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:43.475259 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.475724 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.476900 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.477100 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.477172 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.477485 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.477539 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.477589 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.477701 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.477736 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:33:43.477897 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.474287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40357 (local address 127.10.11.195:59327)
0927 00:33:43.474549 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.474563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.474695 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.475036 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.475038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.475046 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:43.475144 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.475150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.475833 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.475837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.476787 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.476797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.477217 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.477220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.477303 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.477347 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.477617 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.477622 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.477686 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.477756 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260927 00:33:43.484133 12173 raft_consensus.cc:1261] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.484313 12313 raft_consensus.cc:1261] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.484565 12530 consensus_queue.cc:1035] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:43.484807 12532 consensus_queue.cc:1035] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:43.493526 12550 mvcc.cc:205] Tried to move back new op lower bound from 7333761939387289600 to 7333761938039808000. Current Snapshot: MvccSnapshot[applied={T|T < 7333761939387289600}]
I20260927 00:33:43.506980 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.503000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40473 (local address 127.10.11.195:46099)
0927 00:33:43.503121 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:43.503125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:43.503392 (+ 267us) server_negotiation.cc:407] Connection header received
0927 00:33:43.503469 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:43.503472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:43.503520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:43.503598 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:43.504274 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.504977 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.505812 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.506041 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.506164 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.506426 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.506498 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.506576 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:43.506806 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 00:33:43.506853 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:33:43.506995 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:43.502921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:46099 (local address 127.10.11.194:40473)
0927 00:33:43.503300 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:43.503312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:43.503398 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:43.503619 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:43.503621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:43.503629 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:43.504137 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.504148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.505087 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.505092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.505693 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:43.505701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:43.506164 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:43.506167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:43.506248 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:43.506297 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:43.506685 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:43.506692 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:33:43.506790 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:33:43.506863 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260927 00:33:43.532292 12016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:43.532505 12016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:43.535058 12016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Sending DeleteTablet for 3 replicas of tablet aff2bdcdf0b24e309dcc24d40c2d2fda
I20260927 00:33:43.535969 12293 tablet_service.cc:1514] Processing DeleteTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:43 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 00:33:43.536473 12433 tablet_service.cc:1514] Processing DeleteTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:43 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:38018
I20260927 00:33:43.536887 12570 tablet_replica.cc:330] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: stopping tablet replica
I20260927 00:33:43.537153 12570 raft_consensus.cc:2229] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:43.537464 12570 raft_consensus.cc:2258] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:43.538873 12570 ts_tablet_manager.cc:1905] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:43.539459 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet aff2bdcdf0b24e309dcc24d40c2d2fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:43 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:37110
I20260927 00:33:43.540030 12016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:43.540544 12016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:43.542800 12570 ts_tablet_manager.cc:1918] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:43.542909 12570 log.cc:1196] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/aff2bdcdf0b24e309dcc24d40c2d2fda
I20260927 00:33:43.543212 12570 ts_tablet_manager.cc:1939] T aff2bdcdf0b24e309dcc24d40c2d2fda P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting consensus metadata
I20260927 00:33:43.544301 12000 catalog_manager.cc:4841] TS 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099): tablet aff2bdcdf0b24e309dcc24d40c2d2fda (table test-table [id=30f65ff5ce7e4449ab43771387f7672c]) successfully deleted
I20260927 00:33:43.545549 12293 tablet_service.cc:1467] Processing CreateTablet for tablet c91c954914354e3fac33a73b060b0e69 (DEFAULT_TABLE table=test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.545584 12433 tablet_service.cc:1467] Processing CreateTablet for tablet c91c954914354e3fac33a73b060b0e69 (DEFAULT_TABLE table=test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.545811 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91c954914354e3fac33a73b060b0e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.545821 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91c954914354e3fac33a73b060b0e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.548317 12524 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Bootstrap starting.
I20260927 00:33:43.548684 12153 tablet_service.cc:1467] Processing CreateTablet for tablet c91c954914354e3fac33a73b060b0e69 (DEFAULT_TABLE table=test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:43.548952 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91c954914354e3fac33a73b060b0e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:43.549320 12524 tablet_bootstrap.cc:654] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.549942 12523 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Bootstrap starting.
I20260927 00:33:43.551254 12522 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Bootstrap starting.
I20260927 00:33:43.551818 12523 tablet_bootstrap.cc:654] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.551973 12522 tablet_bootstrap.cc:654] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:43.553645 12524 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: No bootstrap required, opened a new log
I20260927 00:33:43.553742 12524 ts_tablet_manager.cc:1397] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 00:33:43.554229 12524 raft_consensus.cc:348] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.554383 12524 raft_consensus.cc:374] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.554430 12524 raft_consensus.cc:729] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.554560 12524 consensus_queue.cc:260] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.554836 12524 ts_tablet_manager.cc:1428] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:43.556375 12522 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: No bootstrap required, opened a new log
I20260927 00:33:43.556468 12522 ts_tablet_manager.cc:1397] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 00:33:43.556905 12522 raft_consensus.cc:348] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.557020 12522 raft_consensus.cc:374] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.557055 12522 raft_consensus.cc:729] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1735e27f24df48c6b498b420865be, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.557180 12522 consensus_queue.cc:260] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.557456 12522 ts_tablet_manager.cc:1428] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:43.558873 12523 tablet_bootstrap.cc:492] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: No bootstrap required, opened a new log
I20260927 00:33:43.558969 12523 ts_tablet_manager.cc:1397] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 00:33:43.559420 12523 raft_consensus.cc:348] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.559540 12523 raft_consensus.cc:374] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:43.559585 12523 raft_consensus.cc:729] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Initialized, Role: FOLLOWER
I20260927 00:33:43.559707 12523 consensus_queue.cc:260] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.559994 12523 ts_tablet_manager.cc:1428] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:43.566299 12571 tablet_replica.cc:330] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: stopping tablet replica
I20260927 00:33:43.566591 12571 raft_consensus.cc:2229] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:43.566784 12571 raft_consensus.cc:2258] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:43.567071 12569 tablet_replica.cc:330] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: stopping tablet replica
I20260927 00:33:43.567327 12569 raft_consensus.cc:2229] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:43.567471 12569 raft_consensus.cc:2258] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:43.569136 12569 ts_tablet_manager.cc:1905] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:43.570075 12571 ts_tablet_manager.cc:1905] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:43.571838 12532 raft_consensus.cc:482] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:43.571920 12532 raft_consensus.cc:504] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.571970 12569 ts_tablet_manager.cc:1918] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:43.572044 12569 log.cc:1196] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/aff2bdcdf0b24e309dcc24d40c2d2fda
I20260927 00:33:43.572261 12532 leader_election.cc:288] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357)
I20260927 00:33:43.572702 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91c954914354e3fac33a73b060b0e69" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f" is_pre_election: true
I20260927 00:33:43.572826 12313 raft_consensus.cc:2454] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:43.573007 12569 ts_tablet_manager.cc:1939] T aff2bdcdf0b24e309dcc24d40c2d2fda P 330bb3c3217440189b5de7e10a574f8f: Deleting consensus metadata
I20260927 00:33:43.573314 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91c954914354e3fac33a73b060b0e69" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be" is_pre_election: true
I20260927 00:33:43.573421 12172 raft_consensus.cc:2454] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:43.573849 12387 leader_election.cc:302] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82; no voters:
I20260927 00:33:43.574164 12532 raft_consensus.cc:2781] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:43.574218 12532 raft_consensus.cc:482] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:43.574179 12003 catalog_manager.cc:4841] TS 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357): tablet aff2bdcdf0b24e309dcc24d40c2d2fda (table test-table [id=30f65ff5ce7e4449ab43771387f7672c]) successfully deleted
I20260927 00:33:43.574254 12532 raft_consensus.cc:3037] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.575676 12571 ts_tablet_manager.cc:1918] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:43.575286 12532 raft_consensus.cc:504] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.575763 12571 log.cc:1196] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/aff2bdcdf0b24e309dcc24d40c2d2fda
I20260927 00:33:43.575978 12532 leader_election.cc:288] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Requested vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357)
I20260927 00:33:43.576083 12571 ts_tablet_manager.cc:1939] T aff2bdcdf0b24e309dcc24d40c2d2fda P a4e1735e27f24df48c6b498b420865be: Deleting consensus metadata
I20260927 00:33:43.576601 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91c954914354e3fac33a73b060b0e69" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be"
I20260927 00:33:43.576699 12172 raft_consensus.cc:3037] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.577178 12001 catalog_manager.cc:4841] TS a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585): tablet aff2bdcdf0b24e309dcc24d40c2d2fda (table test-table [id=30f65ff5ce7e4449ab43771387f7672c]) successfully deleted
I20260927 00:33:43.577406 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91c954914354e3fac33a73b060b0e69" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f"
I20260927 00:33:43.577512 12313 raft_consensus.cc:3037] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:43.577630 12172 raft_consensus.cc:2454] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:43.577946 12385 leader_election.cc:302] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427ae3736ef64b7a8bb4f31a1fb07a82, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:43.578087 12532 raft_consensus.cc:2781] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:43.578176 12532 raft_consensus.cc:686] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Becoming Leader. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Running, Role: LEADER
I20260927 00:33:43.578280 12532 consensus_queue.cc:237] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:43.578473 12313 raft_consensus.cc:2454] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:43.580143 12016 catalog_manager.cc:5495] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195). New cstate: current_term: 1 leader_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260927 00:33:43.600646 12313 raft_consensus.cc:1261] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.601130 12172 raft_consensus.cc:1261] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:43.601327 12532 consensus_queue.cc:1035] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:43.601572 12532 consensus_queue.cc:1035] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:33:43.616182 12551 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:33:45.165693 12016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:45.165830 12016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:45.168113 12016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Sending DeleteTablet for 3 replicas of tablet c91c954914354e3fac33a73b060b0e69
I20260927 00:33:45.168687 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet c91c954914354e3fac33a73b060b0e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:37110
I20260927 00:33:45.168732 12293 tablet_service.cc:1514] Processing DeleteTablet for tablet c91c954914354e3fac33a73b060b0e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 00:33:45.168850 12433 tablet_service.cc:1514] Processing DeleteTablet for tablet c91c954914354e3fac33a73b060b0e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:38018
I20260927 00:33:45.168998 12600 tablet_replica.cc:330] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: stopping tablet replica
I20260927 00:33:45.169584 12600 raft_consensus.cc:2229] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:45.169721 12600 raft_consensus.cc:2258] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:45.170045 12601 tablet_replica.cc:330] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: stopping tablet replica
I20260927 00:33:45.170205 12601 raft_consensus.cc:2229] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:45.170315 12601 raft_consensus.cc:2258] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:45.170502 12602 tablet_replica.cc:330] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: stopping tablet replica
I20260927 00:33:45.170524 12600 ts_tablet_manager.cc:1905] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.170610 12602 raft_consensus.cc:2229] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:45.170790 12602 raft_consensus.cc:2258] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:45.171368 12601 ts_tablet_manager.cc:1905] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.171898 12602 ts_tablet_manager.cc:1905] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.173179 12600 ts_tablet_manager.cc:1918] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:33:45.173243 12600 log.cc:1196] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/c91c954914354e3fac33a73b060b0e69
I20260927 00:33:45.173491 12600 ts_tablet_manager.cc:1939] T c91c954914354e3fac33a73b060b0e69 P a4e1735e27f24df48c6b498b420865be: Deleting consensus metadata
I20260927 00:33:45.174072 12601 ts_tablet_manager.cc:1918] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:33:45.174170 12601 log.cc:1196] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c91c954914354e3fac33a73b060b0e69
I20260927 00:33:45.174455 12601 ts_tablet_manager.cc:1939] T c91c954914354e3fac33a73b060b0e69 P 330bb3c3217440189b5de7e10a574f8f: Deleting consensus metadata
I20260927 00:33:45.174968 12602 ts_tablet_manager.cc:1918] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:33:45.175060 12602 log.cc:1196] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c91c954914354e3fac33a73b060b0e69
I20260927 00:33:45.175112 12003 catalog_manager.cc:4841] TS 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357): tablet c91c954914354e3fac33a73b060b0e69 (table test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]) successfully deleted
I20260927 00:33:45.175382 12602 ts_tablet_manager.cc:1939] T c91c954914354e3fac33a73b060b0e69 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting consensus metadata
I20260927 00:33:45.176087 12000 catalog_manager.cc:4841] TS 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099): tablet c91c954914354e3fac33a73b060b0e69 (table test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]) successfully deleted
I20260927 00:33:45.176097 12001 catalog_manager.cc:4841] TS a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585): tablet c91c954914354e3fac33a73b060b0e69 (table test-table [id=cea261d0c61a4b9993ab6fd1fb46610e]) successfully deleted
Sep 27 00:33:45 dist-test-slave-6gxc krb5kdc[11971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:33:45 dist-test-slave-6gxc krb5kdc[11971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:45.199950 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:45.191011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.10.11.254:44193)
0927 00:33:45.191305 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:45.191317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:45.191354 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:33:45.191527 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:45.191530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:45.191585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:45.191678 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:45.192369 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.192937 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.193853 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.194091 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.194270 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:45.196860 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:45.196863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:45.196865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:45.196885 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:45.198598 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:45.198979 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:45.198982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:45.198983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:45.199056 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:45.199282 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:45.199284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:45.199286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:45.199445 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:45.199543 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:45.199738 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:33:45.199829 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260927 00:33:45.202461 12016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:45.202889 12016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:45.207409 12153 tablet_service.cc:1467] Processing CreateTablet for tablet 633c8b58bdac451a85574ee0eec6748a (DEFAULT_TABLE table=test-table [id=af399a21430241efb057449efb3f5268]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.207450 12293 tablet_service.cc:1467] Processing CreateTablet for tablet 633c8b58bdac451a85574ee0eec6748a (DEFAULT_TABLE table=test-table [id=af399a21430241efb057449efb3f5268]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.207636 12433 tablet_service.cc:1467] Processing CreateTablet for tablet 633c8b58bdac451a85574ee0eec6748a (DEFAULT_TABLE table=test-table [id=af399a21430241efb057449efb3f5268]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.207696 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633c8b58bdac451a85574ee0eec6748a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.207696 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633c8b58bdac451a85574ee0eec6748a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.207870 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633c8b58bdac451a85574ee0eec6748a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.210244 12611 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Bootstrap starting.
I20260927 00:33:45.210460 12612 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Bootstrap starting.
I20260927 00:33:45.210577 12613 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Bootstrap starting.
I20260927 00:33:45.211153 12611 tablet_bootstrap.cc:654] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.211334 12613 tablet_bootstrap.cc:654] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.211436 12612 tablet_bootstrap.cc:654] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.212314 12611 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: No bootstrap required, opened a new log
I20260927 00:33:45.212406 12611 ts_tablet_manager.cc:1397] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:33:45.212426 12613 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: No bootstrap required, opened a new log
I20260927 00:33:45.212507 12613 ts_tablet_manager.cc:1397] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:45.212837 12611 raft_consensus.cc:348] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.212966 12611 raft_consensus.cc:374] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.213011 12611 raft_consensus.cc:729] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.212931 12612 tablet_bootstrap.cc:492] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: No bootstrap required, opened a new log
I20260927 00:33:45.212956 12613 raft_consensus.cc:348] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.213070 12613 raft_consensus.cc:374] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.213110 12613 raft_consensus.cc:729] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.213148 12611 consensus_queue.cc:260] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.213245 12613 consensus_queue.cc:260] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.213312 12612 ts_tablet_manager.cc:1397] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:33:45.213443 12611 ts_tablet_manager.cc:1428] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:33:45.213531 12613 ts_tablet_manager.cc:1428] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:33:45.214181 12612 raft_consensus.cc:348] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.214296 12612 raft_consensus.cc:374] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.214346 12612 raft_consensus.cc:729] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1735e27f24df48c6b498b420865be, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.214458 12612 consensus_queue.cc:260] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.214890 12612 ts_tablet_manager.cc:1428] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:45.367154 12581 raft_consensus.cc:482] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:45.367290 12581 raft_consensus.cc:504] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.367772 12581 leader_election.cc:288] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099)
I20260927 00:33:45.368247 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633c8b58bdac451a85574ee0eec6748a" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" is_pre_election: true
I20260927 00:33:45.368243 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633c8b58bdac451a85574ee0eec6748a" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be" is_pre_election: true
I20260927 00:33:45.368398 12172 raft_consensus.cc:2454] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 0.
I20260927 00:33:45.368398 12439 raft_consensus.cc:2454] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 0.
I20260927 00:33:45.368734 12245 leader_election.cc:302] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:45.368943 12581 raft_consensus.cc:2781] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:45.369019 12581 raft_consensus.cc:482] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:45.369072 12581 raft_consensus.cc:3037] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.370033 12581 raft_consensus.cc:504] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.370380 12581 leader_election.cc:288] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 election: Requested vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099)
I20260927 00:33:45.370812 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633c8b58bdac451a85574ee0eec6748a" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82"
I20260927 00:33:45.370868 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633c8b58bdac451a85574ee0eec6748a" candidate_uuid: "330bb3c3217440189b5de7e10a574f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be"
I20260927 00:33:45.370968 12439 raft_consensus.cc:3037] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.370990 12172 raft_consensus.cc:3037] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.372236 12439 raft_consensus.cc:2454] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 1.
I20260927 00:33:45.372231 12172 raft_consensus.cc:2454] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330bb3c3217440189b5de7e10a574f8f in term 1.
I20260927 00:33:45.372571 12244 leader_election.cc:302] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:45.372763 12581 raft_consensus.cc:2781] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:45.372906 12581 raft_consensus.cc:686] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 1 LEADER]: Becoming Leader. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Running, Role: LEADER
I20260927 00:33:45.373108 12581 consensus_queue.cc:237] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.374912 12016 catalog_manager.cc:5495] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 330bb3c3217440189b5de7e10a574f8f (127.10.11.194). New cstate: current_term: 1 leader_uuid: "330bb3c3217440189b5de7e10a574f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260927 00:33:45.407794 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:45.404478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.10.11.194:40357)
0927 00:33:45.404753 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:45.404769 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:45.404779 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:33:45.404813 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:45.404815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:45.404852 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:45.404941 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:45.405197 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.405738 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.406683 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.406944 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.407053 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:45.407309 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:45.407385 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:45.407487 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:45.407603 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 00:33:45.407651 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260927 00:33:45.415520 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:45.412900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:45161 (local address 127.10.11.194:40357)
0927 00:33:45.413009 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:45.413012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:45.413088 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:33:45.413191 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:45.413193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:45.413230 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:45.413287 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:45.413457 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.413921 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.414727 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.414946 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.415023 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:45.415200 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:45.415252 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:45.415307 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:45.415382 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 00:33:45.415421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":44}
I20260927 00:33:45.415540 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:45.412784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40357 (local address 127.10.11.194:45161)
0927 00:33:45.413019 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:45.413033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:45.413118 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:45.413305 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:45.413306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:45.413312 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:33:45.413384 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:45.413388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.414015 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.414017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:45.414632 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:45.414641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.415036 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.415038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:45.415094 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:45.415114 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:33:45.415321 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:33:45.415325 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:33:45.415371 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 00:33:45.415428 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260927 00:33:45.420313 12172 raft_consensus.cc:1261] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Refusing update from remote peer 330bb3c3217440189b5de7e10a574f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:45.420313 12439 raft_consensus.cc:1261] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Refusing update from remote peer 330bb3c3217440189b5de7e10a574f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:45.420686 12545 consensus_queue.cc:1035] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:45.420825 12581 consensus_queue.cc:1035] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:45.450978 12015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:45.451128 12015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:45.452962 12015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Sending DeleteTablet for 3 replicas of tablet 633c8b58bdac451a85574ee0eec6748a
I20260927 00:33:45.453418 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet 633c8b58bdac451a85574ee0eec6748a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:37110
I20260927 00:33:45.453586 12600 tablet_replica.cc:330] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: stopping tablet replica
I20260927 00:33:45.453689 12600 raft_consensus.cc:2229] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:45.453790 12600 raft_consensus.cc:2258] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:45.453886 12433 tablet_service.cc:1514] Processing DeleteTablet for tablet 633c8b58bdac451a85574ee0eec6748a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:38018
I20260927 00:33:45.454046 12602 tablet_replica.cc:330] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: stopping tablet replica
I20260927 00:33:45.454182 12602 raft_consensus.cc:2229] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:45.454283 12602 raft_consensus.cc:2258] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:45.454298 12292 tablet_service.cc:1514] Processing DeleteTablet for tablet 633c8b58bdac451a85574ee0eec6748a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:45 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 00:33:45.454455 12601 tablet_replica.cc:330] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: stopping tablet replica
I20260927 00:33:45.454574 12601 raft_consensus.cc:2229] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:45.454592 12600 ts_tablet_manager.cc:1905] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.454643 12015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:33:45.454826 12601 raft_consensus.cc:2258] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:33:45.455051 12015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:45.455646 12601 ts_tablet_manager.cc:1905] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.456087 12602 ts_tablet_manager.cc:1905] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:45.457096 12600 ts_tablet_manager.cc:1918] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:45.457160 12600 log.cc:1196] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/633c8b58bdac451a85574ee0eec6748a
I20260927 00:33:45.457427 12600 ts_tablet_manager.cc:1939] T 633c8b58bdac451a85574ee0eec6748a P a4e1735e27f24df48c6b498b420865be: Deleting consensus metadata
I20260927 00:33:45.457969 12601 ts_tablet_manager.cc:1918] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:45.458029 12601 log.cc:1196] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/633c8b58bdac451a85574ee0eec6748a
I20260927 00:33:45.458038 12001 catalog_manager.cc:4841] TS a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585): tablet 633c8b58bdac451a85574ee0eec6748a (table test-table [id=af399a21430241efb057449efb3f5268]) successfully deleted
I20260927 00:33:45.458289 12601 ts_tablet_manager.cc:1939] T 633c8b58bdac451a85574ee0eec6748a P 330bb3c3217440189b5de7e10a574f8f: Deleting consensus metadata
I20260927 00:33:45.458580 12602 ts_tablet_manager.cc:1918] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:33:45.458650 12602 log.cc:1196] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/633c8b58bdac451a85574ee0eec6748a
I20260927 00:33:45.458842 12003 catalog_manager.cc:4841] TS 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357): tablet 633c8b58bdac451a85574ee0eec6748a (table test-table [id=af399a21430241efb057449efb3f5268]) successfully deleted
I20260927 00:33:45.458920 12602 ts_tablet_manager.cc:1939] T 633c8b58bdac451a85574ee0eec6748a P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting consensus metadata
I20260927 00:33:45.459887 12000 catalog_manager.cc:4841] TS 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099): tablet 633c8b58bdac451a85574ee0eec6748a (table test-table [id=af399a21430241efb057449efb3f5268]) successfully deleted
I20260927 00:33:45.462308 12153 tablet_service.cc:1467] Processing CreateTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 (DEFAULT_TABLE table=test-table [id=12a77fb3138843f98ced41c2aa095d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.462431 12292 tablet_service.cc:1467] Processing CreateTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 (DEFAULT_TABLE table=test-table [id=12a77fb3138843f98ced41c2aa095d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.462539 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a5c14843574b99b93a8a9eab4c6741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.462519 12433 tablet_service.cc:1467] Processing CreateTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 (DEFAULT_TABLE table=test-table [id=12a77fb3138843f98ced41c2aa095d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:45.462644 12292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a5c14843574b99b93a8a9eab4c6741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.462726 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a5c14843574b99b93a8a9eab4c6741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:45.464680 12611 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Bootstrap starting.
I20260927 00:33:45.464713 12612 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: Bootstrap starting.
I20260927 00:33:45.465080 12613 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Bootstrap starting.
I20260927 00:33:45.465394 12611 tablet_bootstrap.cc:654] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.465591 12612 tablet_bootstrap.cc:654] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.465874 12613 tablet_bootstrap.cc:654] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:45.466917 12612 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: No bootstrap required, opened a new log
I20260927 00:33:45.467010 12612 ts_tablet_manager.cc:1397] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:45.467196 12613 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: No bootstrap required, opened a new log
I20260927 00:33:45.467275 12613 ts_tablet_manager.cc:1397] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:45.467465 12612 raft_consensus.cc:348] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.467576 12612 raft_consensus.cc:374] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.467617 12612 raft_consensus.cc:729] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1735e27f24df48c6b498b420865be, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.467654 12613 raft_consensus.cc:348] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.467761 12613 raft_consensus.cc:374] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.467746 12612 consensus_queue.cc:260] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.467804 12613 raft_consensus.cc:729] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bb3c3217440189b5de7e10a574f8f, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.467911 12613 consensus_queue.cc:260] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.468021 12612 ts_tablet_manager.cc:1428] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:45.468159 12613 ts_tablet_manager.cc:1428] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:45.469600 12611 tablet_bootstrap.cc:492] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: No bootstrap required, opened a new log
I20260927 00:33:45.469683 12611 ts_tablet_manager.cc:1397] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 00:33:45.470045 12611 raft_consensus.cc:348] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.470180 12611 raft_consensus.cc:374] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:45.470232 12611 raft_consensus.cc:729] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Initialized, Role: FOLLOWER
I20260927 00:33:45.470346 12611 consensus_queue.cc:260] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.470618 12611 ts_tablet_manager.cc:1428] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:33:45.499214 12591 raft_consensus.cc:482] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:45.499325 12591 raft_consensus.cc:504] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.499692 12591 leader_election.cc:288] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357)
I20260927 00:33:45.500100 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a5c14843574b99b93a8a9eab4c6741" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f" is_pre_election: true
I20260927 00:33:45.500099 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a5c14843574b99b93a8a9eab4c6741" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be" is_pre_election: true
I20260927 00:33:45.500249 12313 raft_consensus.cc:2454] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:45.500274 12172 raft_consensus.cc:2454] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 0.
I20260927 00:33:45.500574 12387 leader_election.cc:302] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82, a4e1735e27f24df48c6b498b420865be; no voters:
I20260927 00:33:45.500773 12591 raft_consensus.cc:2781] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:45.500839 12591 raft_consensus.cc:482] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:45.500877 12591 raft_consensus.cc:3037] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.501993 12591 raft_consensus.cc:504] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.502351 12591 leader_election.cc:288] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Requested vote from peers a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585), 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357)
I20260927 00:33:45.502647 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a5c14843574b99b93a8a9eab4c6741" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e1735e27f24df48c6b498b420865be"
I20260927 00:33:45.502662 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a5c14843574b99b93a8a9eab4c6741" candidate_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330bb3c3217440189b5de7e10a574f8f"
I20260927 00:33:45.502784 12313 raft_consensus.cc:3037] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.502784 12172 raft_consensus.cc:3037] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:45.503782 12172 raft_consensus.cc:2454] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:45.503815 12313 raft_consensus.cc:2454] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427ae3736ef64b7a8bb4f31a1fb07a82 in term 1.
I20260927 00:33:45.504055 12387 leader_election.cc:302] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330bb3c3217440189b5de7e10a574f8f, 427ae3736ef64b7a8bb4f31a1fb07a82; no voters:
I20260927 00:33:45.504287 12591 raft_consensus.cc:2781] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:45.504386 12591 raft_consensus.cc:686] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Becoming Leader. State: Replica: 427ae3736ef64b7a8bb4f31a1fb07a82, State: Running, Role: LEADER
I20260927 00:33:45.504516 12591 consensus_queue.cc:237] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } }
I20260927 00:33:45.506151 12015 catalog_manager.cc:5495] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195). New cstate: current_term: 1 leader_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427ae3736ef64b7a8bb4f31a1fb07a82" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260927 00:33:45.529537 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:45.526088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.10.11.195:46099)
0927 00:33:45.526387 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:45.526394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:45.526406 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:33:45.526441 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:45.526443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:45.526479 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:45.526550 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:45.526815 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.527330 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.528473 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:45.528691 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:45.528768 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:45.529065 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:45.529133 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:45.529199 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:45.529302 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 00:33:45.529350 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":119,"threads_started":1}
I20260927 00:33:45.538295 12172 raft_consensus.cc:1261] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:45.538647 12313 raft_consensus.cc:1261] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Refusing update from remote peer 427ae3736ef64b7a8bb4f31a1fb07a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:45.538878 12591 consensus_queue.cc:1035] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e1735e27f24df48c6b498b420865be" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:45.539129 12591 consensus_queue.cc:1035] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "330bb3c3217440189b5de7e10a574f8f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:33:45.553674 12582 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:33:47.104339 12015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:47.104487 12015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:47.106758 12015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 000e832767ce4e92b39cc4099fca1da4: Sending DeleteTablet for 3 replicas of tablet a9a5c14843574b99b93a8a9eab4c6741
I20260927 00:33:47.107270 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:47 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:37110
I20260927 00:33:47.107342 12433 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:47 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:38018
I20260927 00:33:47.107370 12292 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a5c14843574b99b93a8a9eab4c6741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:47 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 00:33:47.107717 12659 tablet_replica.cc:330] T a9a5c14843574b99b93a8a9eab4c6741 P a4e1735e27f24df48c6b498b420865be: stopping tablet replica
I20260927 00:33:47.107733 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12088
I20260927 00:33:47.108433 12660 tablet_replica.cc:330] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: stopping tablet replica
I20260927 00:33:47.108455 12661 tablet_replica.cc:330] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: stopping tablet replica
I20260927 00:33:47.108582 12660 raft_consensus.cc:2229] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:33:47.108600 12661 raft_consensus.cc:2229] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:47.108709 12661 raft_consensus.cc:2258] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:47.108776 12660 raft_consensus.cc:2258] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:47.109647 12661 ts_tablet_manager.cc:1905] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:47.113164 12661 ts_tablet_manager.cc:1918] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:33:47.113258 12661 log.cc:1196] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/a9a5c14843574b99b93a8a9eab4c6741
I20260927 00:33:47.113575 12661 ts_tablet_manager.cc:1939] T a9a5c14843574b99b93a8a9eab4c6741 P 330bb3c3217440189b5de7e10a574f8f: Deleting consensus metadata
I20260927 00:33:47.113679 12660 ts_tablet_manager.cc:1905] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:47.114382 12003 catalog_manager.cc:4841] TS 330bb3c3217440189b5de7e10a574f8f (127.10.11.194:40357): tablet a9a5c14843574b99b93a8a9eab4c6741 (table test-table [id=12a77fb3138843f98ced41c2aa095d41]) successfully deleted
W20260927 00:33:47.115658 12001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:39585 (error 108)
I20260927 00:33:47.115934 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12228
I20260927 00:33:47.116492 12660 ts_tablet_manager.cc:1918] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:33:47.116567 12660 log.cc:1196] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/a9a5c14843574b99b93a8a9eab4c6741
I20260927 00:33:47.116811 12660 ts_tablet_manager.cc:1939] T a9a5c14843574b99b93a8a9eab4c6741 P 427ae3736ef64b7a8bb4f31a1fb07a82: Deleting consensus metadata
I20260927 00:33:47.117324 12000 catalog_manager.cc:4841] TS 427ae3736ef64b7a8bb4f31a1fb07a82 (127.10.11.195:46099): tablet a9a5c14843574b99b93a8a9eab4c6741 (table test-table [id=12a77fb3138843f98ced41c2aa095d41]) successfully deleted
I20260927 00:33:47.117974 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:47.117489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39585 (local address 127.0.0.1:37118)
0927 00:33:47.117754 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:47.117854 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
W20260927 00:33:47.118125 12001 catalog_manager.cc:4584] TS a4e1735e27f24df48c6b498b420865be (127.10.11.193:39585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9a5c14843574b99b93a8a9eab4c6741: Network error: Client connection negotiation failed: client connection to 127.10.11.193:39585: connect: Connection refused (error 111)
I20260927 00:33:47.123117 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12368
I20260927 00:33:47.129658 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 11984
2026-09-27T00:33:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6198 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:47 dist-test-slave-6gxc krb5kdc[12666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:47 dist-test-slave-6gxc krb5kdc[12666](info): set up 2 sockets
Sep 27 00:33:47 dist-test-slave-6gxc krb5kdc[12666](info): commencing operation
krb5kdc: starting...
W20260927 00:33:48.162251 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:48.276490 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41949
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:48.382237 12679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:48.382469 12679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:48.382507 12679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:48.386533 12679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:48.386581 12679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:48.386600 12679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:48.386617 12679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:48.386664 12679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:48.391789 12679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:48.392776 12679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:48.393707 12679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:48.402215 12687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:48.402215 12685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:48.402346 12679 server_base.cc:1034] running on GCE node
W20260927 00:33:48.402302 12684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:48.403038 12679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:48.403996 12679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:48.405179 12679 hybrid_clock.cc:648] HybridClock initialized: now 1790469228405166 us; error 30 us; skew 500 ppm
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:48.411662 12679 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:48.412472 12679 webserver.cc:460] Webserver started at http://127.10.11.254:36535/ using document root <none> and password file <none>
I20260927 00:33:48.413012 12679 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:48.413097 12679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:48.413358 12679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:48.415323 12679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "b2166c8669ea4bc0858da4685ea88c9a"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "37c7f3f0324ab563c58266c4fd2ffae2"
server_key_iv: "eb109739616d8a3aacdf498d82d4d2cf"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.415861 12679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "b2166c8669ea4bc0858da4685ea88c9a"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "37c7f3f0324ab563c58266c4fd2ffae2"
server_key_iv: "eb109739616d8a3aacdf498d82d4d2cf"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.419275 12679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:48.421530 12694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.422835 12679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:48.422981 12679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "b2166c8669ea4bc0858da4685ea88c9a"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "37c7f3f0324ab563c58266c4fd2ffae2"
server_key_iv: "eb109739616d8a3aacdf498d82d4d2cf"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.423089 12679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:48.436621 12679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:48.439404 12679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:48.439581 12679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:48.446102 12679 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:41949
I20260927 00:33:48.446142 12746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:41949 every 8 connection(s)
I20260927 00:33:48.446996 12679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:48.449455 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:48.451757 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12679
I20260927 00:33:48.451864 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:48.452145 10287 external_mini_cluster.cc:1428] Setting key 1dedd9da18609f49efa84ceed705d0c8
I20260927 00:33:48.455256 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: Bootstrap starting.
I20260927 00:33:48.457284 12747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:48.457962 12747 log.cc:824] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:48.460098 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: No bootstrap required, opened a new log
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:48.462911 12747 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } }
I20260927 00:33:48.463128 12747 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:48.463208 12747 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2166c8669ea4bc0858da4685ea88c9a, State: Initialized, Role: FOLLOWER
I20260927 00:33:48.463678 12747 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } }
I20260927 00:33:48.463822 12747 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:48.463902 12747 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:48.463996 12747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:48.464762 12747 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } }
I20260927 00:33:48.465076 12747 leader_election.cc:302] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2166c8669ea4bc0858da4685ea88c9a; no voters:
I20260927 00:33:48.465384 12747 leader_election.cc:288] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:48.465442 12752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:48.465600 12752 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [term 1 LEADER]: Becoming Leader. State: Replica: b2166c8669ea4bc0858da4685ea88c9a, State: Running, Role: LEADER
I20260927 00:33:48.465919 12752 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } }
I20260927 00:33:48.466538 12747 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:48.467062 12752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2166c8669ea4bc0858da4685ea88c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } } }
I20260927 00:33:48.467541 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:48.453519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.10.11.254:41949)
0927 00:33:48.453919 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:48.453933 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:48.453964 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:33:48.454634 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:48.454649 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:48.454963 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:48.455278 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:48.456060 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.457005 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.457897 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.458489 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.458660 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:48.461245 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:48.461250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:48.461261 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:48.461285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:48.463976 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.464466 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.464472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.464478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.464611 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.464921 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.464926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.464928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.465387 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:48.465622 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:48.465962 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 00:33:48.466476 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20260927 00:33:48.469412 12752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:48.467700 12753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2166c8669ea4bc0858da4685ea88c9a. Latest consensus state: current_term: 1 leader_uuid: "b2166c8669ea4bc0858da4685ea88c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2166c8669ea4bc0858da4685ea88c9a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41949 } } }
I20260927 00:33:48.469687 12753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a [sys.catalog]: This master's current role is: LEADER
W20260927 00:33:48.471515 12766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:48.471580 12766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:48.471666 12768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:48.472399 12768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:48.477483 12768 catalog_manager.cc:1347] Generated new cluster ID: ac775a081e144e66a18bad31b47e7410
I20260927 00:33:48.477546 12768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:48.484948 12768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:48.486038 12768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:48.497543 12768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2166c8669ea4bc0858da4685ea88c9a: Generated new TSK 0
I20260927 00:33:48.498217 12768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:48.526786 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:48.631951 12773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:48.632170 12773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:48.632216 12773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:48.636500 12773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:48.636560 12773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:48.636665 12773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:48.641712 12773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:48.642652 12773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:48.643590 12773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:48.652529 12778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:48.652526 12779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:48.652530 12781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:48.652913 12773 server_base.cc:1034] running on GCE node
I20260927 00:33:48.653257 12773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:48.653784 12773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:48.654934 12773 hybrid_clock.cc:648] HybridClock initialized: now 1790469228654917 us; error 28 us; skew 500 ppm
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:48.657470 12773 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:48.658346 12773 webserver.cc:460] Webserver started at http://127.10.11.193:36721/ using document root <none> and password file <none>
I20260927 00:33:48.658878 12773 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:48.658963 12773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:48.659173 12773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:48.661010 12773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "906b6e480a08490892275244e290323c"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "81f80196aa8c8bc86ce07af9ffa95150"
server_key_iv: "0405368ee0c2edf12ede8520c476c262"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.661564 12773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "906b6e480a08490892275244e290323c"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "81f80196aa8c8bc86ce07af9ffa95150"
server_key_iv: "0405368ee0c2edf12ede8520c476c262"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.664991 12773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:33:48.667346 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.668474 12773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:48.668596 12773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "906b6e480a08490892275244e290323c"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "81f80196aa8c8bc86ce07af9ffa95150"
server_key_iv: "0405368ee0c2edf12ede8520c476c262"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.668733 12773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:48.685928 12773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:48.688953 12773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:48.689141 12773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:48.689638 12773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:48.690382 12773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:48.690451 12773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.690516 12773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:48.690558 12773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.699391 12773 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:42513
I20260927 00:33:48.699414 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:42513 every 8 connection(s)
I20260927 00:33:48.700165 12773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:48.702389 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12773
I20260927 00:33:48.702512 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:48.702813 10287 external_mini_cluster.cc:1428] Setting key abd22bbc80a6a1e246ca50d3d5837b7a
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:33:48.714342 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:48.701810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52193 (local address 127.10.11.254:41949)
0927 00:33:48.701927 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:48.701934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:48.702992 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 00:33:48.703994 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:48.703997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:48.704063 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:48.704151 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:48.705246 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.706030 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.707485 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.707928 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.708052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:48.710634 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:48.710640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:48.710645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:48.710675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:48.712335 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.712862 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.712865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.712866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.712939 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.713343 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.713346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.713347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.713511 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:48.713588 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:48.714082 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 00:33:48.714225 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:33:48.715111 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:48.702182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:41949 (local address 127.10.11.193:52193)
0927 00:33:48.702843 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:48.702889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:48.703793 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:48.704331 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:48.704340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:48.704665 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:48.705104 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:48.705116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.706184 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.706190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:48.707351 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:48.707361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.708091 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.708095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:48.708223 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:48.708771 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:48.708787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:48.710479 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:48.712488 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:48.712492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:48.712502 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:48.712731 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:48.713046 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:48.713048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:48.713050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:48.713239 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:48.713619 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:48.713623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:48.713825 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 00:33:48.714021 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 00:33:48.714300 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":120,"threads_started":1}
I20260927 00:33:48.716172 12902 heartbeater.cc:347] Connected to a master server at 127.10.11.254:41949
I20260927 00:33:48.716418 12902 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:48.716945 12902 heartbeater.cc:511] Master 127.10.11.254:41949 requested a full tablet report, sending...
I20260927 00:33:48.718365 12710 ts_manager.cc:194] Registered new tserver with Master: 906b6e480a08490892275244e290323c (127.10.11.193:42513)
I20260927 00:33:48.719405 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:52193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:48.728930 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:48.834676 12907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:48.834908 12907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:48.834946 12907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:48.838851 12907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:48.838900 12907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:48.838954 12907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:48.843947 12907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:48.844779 12907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:48.845638 12907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:33:48.854699 12907 server_base.cc:1034] running on GCE node
W20260927 00:33:48.854637 12913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:48.854652 12912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:48.854661 12915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:48.855293 12907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:48.855885 12907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:48.857036 12907 hybrid_clock.cc:648] HybridClock initialized: now 1790469228857022 us; error 26 us; skew 500 ppm
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:48.859725 12907 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:48.860510 12907 webserver.cc:460] Webserver started at http://127.10.11.194:40643/ using document root <none> and password file <none>
I20260927 00:33:48.861037 12907 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:48.861099 12907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:48.861294 12907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:48.863216 12907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "84942c4bcb68450ab28383583e04560b"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "63b13085f91eb1c94752902244e9fb5a"
server_key_iv: "f50da23021f2d459c5e0b089bf0bd73a"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.863803 12907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "84942c4bcb68450ab28383583e04560b"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "63b13085f91eb1c94752902244e9fb5a"
server_key_iv: "f50da23021f2d459c5e0b089bf0bd73a"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.867333 12907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:48.869625 12922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.870795 12907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:33:48.870896 12907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "84942c4bcb68450ab28383583e04560b"
format_stamp: "Formatted at 2026-09-27 00:33:48 on dist-test-slave-6gxc"
server_key: "63b13085f91eb1c94752902244e9fb5a"
server_key_iv: "f50da23021f2d459c5e0b089bf0bd73a"
server_key_version: "encryptionkey@0"
I20260927 00:33:48.870970 12907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:48.898414 12907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:48.901152 12907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:48.901288 12907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:48.901757 12907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:48.902594 12907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:48.902639 12907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.902678 12907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:48.902700 12907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:48.911634 12907 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:37069
I20260927 00:33:48.911701 13035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:37069 every 8 connection(s)
I20260927 00:33:48.912477 12907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:48.914582 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 12907
I20260927 00:33:48.914693 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:48.915019 10287 external_mini_cluster.cc:1428] Setting key 499b1aafd3349be36d78ba086ec3d170
Sep 27 00:33:48 dist-test-slave-6gxc krb5kdc[12666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469228, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:33:48.926599 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:48.914194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:39159 (local address 127.10.11.254:41949)
0927 00:33:48.914318 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:48.914321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:48.915830 (+ 1509us) server_negotiation.cc:407] Connection header received
0927 00:33:48.916792 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:48.916794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:48.916834 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:48.916904 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:48.918429 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.919288 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.920406 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.920638 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.920743 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:48.923078 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:48.923081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:48.923083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:48.923103 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:48.924494 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.925081 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.925084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.925086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.925162 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:48.925491 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:48.925494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:48.925496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:48.925716 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:48.925815 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:48.926325 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 00:33:48.926446 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:33:48.927479 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:48.914639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:41949 (local address 127.10.11.194:39159)
0927 00:33:48.915701 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:48.915750 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:48.916589 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:48.917072 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:48.917084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:48.917602 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:48.918299 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:48.918316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.919432 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.919440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:48.920277 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:48.920285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:48.920774 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:48.920778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:48.920941 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:48.921527 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:48.921541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:48.922912 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:48.924622 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:48.924627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:48.924640 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:48.924975 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:48.925258 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:48.925261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:48.925263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:48.925393 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:48.925825 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:48.925834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:48.926056 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:33:48.926269 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:33:48.926598 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":682,"threads_started":1}
I20260927 00:33:48.928628 13036 heartbeater.cc:347] Connected to a master server at 127.10.11.254:41949
I20260927 00:33:48.928892 13036 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:48.929355 13036 heartbeater.cc:511] Master 127.10.11.254:41949 requested a full tablet report, sending...
I20260927 00:33:48.930315 12710 ts_manager.cc:194] Registered new tserver with Master: 84942c4bcb68450ab28383583e04560b (127.10.11.194:37069)
I20260927 00:33:48.931002 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:39159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:48.943390 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:49.049189 13041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:49.049418 13041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:49.049458 13041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:49.053329 13041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:49.053385 13041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:49.053442 13041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:49.058476 13041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:49.059418 13041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:49.060313 13041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:49.070683 13046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:49.070676 13047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:49.070695 13049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:49.070936 13041 server_base.cc:1034] running on GCE node
I20260927 00:33:49.071282 13041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:49.071952 13041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:49.073127 13041 hybrid_clock.cc:648] HybridClock initialized: now 1790469229073115 us; error 23 us; skew 500 ppm
Sep 27 00:33:49 dist-test-slave-6gxc krb5kdc[12666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469229, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:49.075831 13041 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:49.076659 13041 webserver.cc:460] Webserver started at http://127.10.11.195:46511/ using document root <none> and password file <none>
I20260927 00:33:49.077183 13041 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:49.077270 13041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:49.077485 13041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:49.079463 13041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "48aff0d10ac74fd9990cb8e09afbdcf5"
format_stamp: "Formatted at 2026-09-27 00:33:49 on dist-test-slave-6gxc"
server_key: "ed5d9015a11220ef2527be8800f89c7f"
server_key_iv: "1fe2b73f7acfe5890aca510ffab0de88"
server_key_version: "encryptionkey@0"
I20260927 00:33:49.080009 13041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "48aff0d10ac74fd9990cb8e09afbdcf5"
format_stamp: "Formatted at 2026-09-27 00:33:49 on dist-test-slave-6gxc"
server_key: "ed5d9015a11220ef2527be8800f89c7f"
server_key_iv: "1fe2b73f7acfe5890aca510ffab0de88"
server_key_version: "encryptionkey@0"
I20260927 00:33:49.083536 13041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:33:49.085806 13056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:49.086999 13041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:49.087127 13041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "48aff0d10ac74fd9990cb8e09afbdcf5"
format_stamp: "Formatted at 2026-09-27 00:33:49 on dist-test-slave-6gxc"
server_key: "ed5d9015a11220ef2527be8800f89c7f"
server_key_iv: "1fe2b73f7acfe5890aca510ffab0de88"
server_key_version: "encryptionkey@0"
I20260927 00:33:49.087229 13041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:49.099723 13041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:49.102506 13041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:49.102681 13041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:49.103193 13041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:49.103940 13041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:49.104008 13041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:49.104071 13041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:49.104120 13041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:49.112514 13041 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:43783
I20260927 00:33:49.112538 13169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:43783 every 8 connection(s)
I20260927 00:33:49.113294 13041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:49.118743 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13041
I20260927 00:33:49.118858 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:49.119128 10287 external_mini_cluster.cc:1428] Setting key c777ba3f8b380ac50f0d94a22ad2b655
Sep 27 00:33:49 dist-test-slave-6gxc krb5kdc[12666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469229, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:49.125782 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:49.115016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:44927 (local address 127.10.11.254:41949)
0927 00:33:49.115174 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:49.115179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:49.116098 (+ 919us) server_negotiation.cc:407] Connection header received
0927 00:33:49.116850 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:49.116853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:49.116911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:49.116983 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:49.117995 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:49.118531 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:49.119969 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:49.120185 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:49.120292 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:49.122509 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:49.122512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:49.122520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:49.122540 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:49.123953 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:49.124433 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:49.124437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:49.124439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:49.124512 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:49.124824 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:49.124828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:49.124830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:49.124993 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:49.125091 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:49.125521 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 00:33:49.125628 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:33:49.126569 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:49.115362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:41949 (local address 127.10.11.195:44927)
0927 00:33:49.115965 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:49.116005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:49.116691 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:49.117128 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:49.117134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:49.117442 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:49.117864 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:49.117876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:49.118643 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:49.118647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:49.119851 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:49.119860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:49.120321 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:49.120325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:49.120484 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:49.121004 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:49.121019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:49.122364 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:49.124078 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:49.124084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:49.124096 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:49.124330 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:49.124605 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:49.124608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:49.124610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:49.124732 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:49.125088 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:49.125094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:49.125306 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 00:33:49.125471 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 00:33:49.125745 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":101,"threads_started":1}
I20260927 00:33:49.127566 13170 heartbeater.cc:347] Connected to a master server at 127.10.11.254:41949
I20260927 00:33:49.127774 13170 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:49.128193 13170 heartbeater.cc:511] Master 127.10.11.254:41949 requested a full tablet report, sending...
I20260927 00:33:49.129076 12710 ts_manager.cc:194] Registered new tserver with Master: 48aff0d10ac74fd9990cb8e09afbdcf5 (127.10.11.195:43783)
I20260927 00:33:49.129566 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:44927
I20260927 00:33:49.132444 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:33:49.143994 12750 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:33:49.142664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.10.11.254:41949)
0927 00:33:49.142785 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:49.142790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:49.142996 (+ 206us) server_negotiation.cc:407] Connection header received
0927 00:33:49.143146 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:49.143149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:49.143190 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:49.143284 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:49.143906 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46974: BlockingRecv error: recv got EOF from 127.0.0.1:46974 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:33:49.145136 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12773
I20260927 00:33:49.150317 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12907
I20260927 00:33:49.155447 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13041
I20260927 00:33:49.160140 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 12679
2026-09-27T00:33:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2029 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:49 dist-test-slave-6gxc krb5kdc[13182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:49 dist-test-slave-6gxc krb5kdc[13182](info): set up 2 sockets
Sep 27 00:33:49 dist-test-slave-6gxc krb5kdc[13182](info): commencing operation
krb5kdc: starting...
W20260927 00:33:50.186494 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:50.298321 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36253
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:50.400486 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:50.400703 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:50.400739 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:50.404685 13195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:50.404732 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:50.404752 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:50.404778 13195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:50.404793 13195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:50.409726 13195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:50.410607 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:50.411445 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:50.418938 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:50.419025 13195 server_base.cc:1034] running on GCE node
W20260927 00:33:50.418938 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:50.419023 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:50.419668 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:50.420639 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:50.421852 13195 hybrid_clock.cc:648] HybridClock initialized: now 1790469230421834 us; error 27 us; skew 500 ppm
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469230, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:50.424540 13195 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:50.425318 13195 webserver.cc:460] Webserver started at http://127.10.11.254:37681/ using document root <none> and password file <none>
I20260927 00:33:50.425837 13195 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:50.425925 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:50.426182 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:50.428006 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "d882a5258ccf4fce85d3b5d378a96303"
format_stamp: "Formatted at 2026-09-27 00:33:50 on dist-test-slave-6gxc"
server_key: "79d0a196d9a9f183cfbafe956746c818"
server_key_iv: "f1bfca3e4137430509188a563a668173"
server_key_version: "encryptionkey@0"
I20260927 00:33:50.428530 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "d882a5258ccf4fce85d3b5d378a96303"
format_stamp: "Formatted at 2026-09-27 00:33:50 on dist-test-slave-6gxc"
server_key: "79d0a196d9a9f183cfbafe956746c818"
server_key_iv: "f1bfca3e4137430509188a563a668173"
server_key_version: "encryptionkey@0"
I20260927 00:33:50.431968 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:33:50.434319 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:50.435463 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:50.435592 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "d882a5258ccf4fce85d3b5d378a96303"
format_stamp: "Formatted at 2026-09-27 00:33:50 on dist-test-slave-6gxc"
server_key: "79d0a196d9a9f183cfbafe956746c818"
server_key_iv: "f1bfca3e4137430509188a563a668173"
server_key_version: "encryptionkey@0"
I20260927 00:33:50.435691 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:50.458058 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:50.461154 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:50.461341 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:50.467672 13195 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:36253
I20260927 00:33:50.467947 13262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:36253 every 8 connection(s)
I20260927 00:33:50.468565 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:50.471405 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:50.473058 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13195
I20260927 00:33:50.473186 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:50.473454 10287 external_mini_cluster.cc:1428] Setting key 53fa8bbcf383dba9e590d4bf4d6ce232
I20260927 00:33:50.477998 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303: Bootstrap starting.
I20260927 00:33:50.480556 13263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:50.481271 13263 log.cc:824] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:50.484007 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303: No bootstrap required, opened a new log
I20260927 00:33:50.487587 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:50.474672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.10.11.254:36253)
0927 00:33:50.475122 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:50.475142 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:50.475182 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:33:50.476011 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:50.476032 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:50.476425 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:50.476789 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:50.477433 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:50.478870 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:50.479724 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:50.480388 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:50.480596 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:50.483156 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:50.483162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:50.483173 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:50.483213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:50.484880 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:50.485347 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:50.485351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:50.485355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:50.485463 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:50.485772 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:50.485774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:50.485775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:50.486174 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:50.486332 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:50.486611 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:33:50.486918 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
I20260927 00:33:50.487880 13263 raft_consensus.cc:348] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } }
I20260927 00:33:50.488098 13263 raft_consensus.cc:374] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:50.488173 13263 raft_consensus.cc:729] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d882a5258ccf4fce85d3b5d378a96303, State: Initialized, Role: FOLLOWER
I20260927 00:33:50.488572 13263 consensus_queue.cc:260] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } }
I20260927 00:33:50.488713 13263 raft_consensus.cc:388] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:50.488776 13263 raft_consensus.cc:482] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:50.488890 13263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:50.489807 13263 raft_consensus.cc:504] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } }
I20260927 00:33:50.490217 13263 leader_election.cc:302] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d882a5258ccf4fce85d3b5d378a96303; no voters:
I20260927 00:33:50.490531 13263 leader_election.cc:288] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:50.490581 13268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:50.490620 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:33:50.490819 13268 raft_consensus.cc:686] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [term 1 LEADER]: Becoming Leader. State: Replica: d882a5258ccf4fce85d3b5d378a96303, State: Running, Role: LEADER
I20260927 00:33:50.491169 13268 consensus_queue.cc:237] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } }
I20260927 00:33:50.491658 13263 sys_catalog.cc:564] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:50.492314 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d882a5258ccf4fce85d3b5d378a96303. Latest consensus state: current_term: 1 leader_uuid: "d882a5258ccf4fce85d3b5d378a96303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } } }
I20260927 00:33:50.493083 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:50.492971 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d882a5258ccf4fce85d3b5d378a96303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882a5258ccf4fce85d3b5d378a96303" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36253 } } }
I20260927 00:33:50.493417 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:50.493489 13278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:50.496723 13278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:50.502218 13278 catalog_manager.cc:1347] Generated new cluster ID: 9652c3de59d64a47811946fb8e927934
I20260927 00:33:50.502276 13278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:50.506973 13278 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:50.507799 13278 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:50.514894 13278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d882a5258ccf4fce85d3b5d378a96303: Generated new TSK 0
I20260927 00:33:50.515575 13278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:33:50.517180 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:50.511780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.10.11.254:36253)
0927 00:33:50.511874 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:50.511879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:50.512140 (+ 261us) server_negotiation.cc:407] Connection header received
0927 00:33:50.512254 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:50.512257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:50.512308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:50.512401 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:50.512918 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:50.513742 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:50.514883 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:50.515235 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:50.515339 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:50.516245 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:50.516249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:50.516252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:50.516736 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:50.516778 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:50.516797 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:50.516884 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:50.516985 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 00:33:50.517045 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":33}
I20260927 00:33:50.521543 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13195
2026-09-27T00:33:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1359 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13296](info): set up 2 sockets
Sep 27 00:33:50 dist-test-slave-6gxc krb5kdc[13296](info): commencing operation
krb5kdc: starting...
W20260927 00:33:51.550566 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:51 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:51.663151 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:37405
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:51.775724 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:51.776110 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:51.776187 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:51.782934 13309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:51.783007 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:51.783046 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:51.783084 13309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:51.783107 13309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:51.791497 13309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:51.792793 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:51.794045 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:51.802398 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:51.802404 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:51.802404 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:51.802604 13309 server_base.cc:1034] running on GCE node
I20260927 00:33:51.803308 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:51.804272 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:51.805426 13309 hybrid_clock.cc:648] HybridClock initialized: now 1790469231805415 us; error 26 us; skew 500 ppm
Sep 27 00:33:51 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469231, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:51.808149 13309 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:51.809181 13309 webserver.cc:460] Webserver started at http://127.10.11.254:42165/ using document root <none> and password file <none>
I20260927 00:33:51.809708 13309 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:51.809795 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:51.809999 13309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:51.811968 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "339766ca302e4576b46d68da9949aab6"
format_stamp: "Formatted at 2026-09-27 00:33:51 on dist-test-slave-6gxc"
server_key: "19dca61cc79a4f731b9f30a31a5f3ee9"
server_key_iv: "e800be21bbac0d60cf1dff0f2834eaaa"
server_key_version: "encryptionkey@0"
I20260927 00:33:51.812503 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "339766ca302e4576b46d68da9949aab6"
format_stamp: "Formatted at 2026-09-27 00:33:51 on dist-test-slave-6gxc"
server_key: "19dca61cc79a4f731b9f30a31a5f3ee9"
server_key_iv: "e800be21bbac0d60cf1dff0f2834eaaa"
server_key_version: "encryptionkey@0"
I20260927 00:33:51.815953 13309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:51.818259 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:51.819427 13309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:51.819572 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "339766ca302e4576b46d68da9949aab6"
format_stamp: "Formatted at 2026-09-27 00:33:51 on dist-test-slave-6gxc"
server_key: "19dca61cc79a4f731b9f30a31a5f3ee9"
server_key_iv: "e800be21bbac0d60cf1dff0f2834eaaa"
server_key_version: "encryptionkey@0"
I20260927 00:33:51.819674 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:51.849611 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:51.852841 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:51.853039 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:51.859767 13309 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:37405
I20260927 00:33:51.860086 13376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:37405 every 8 connection(s)
I20260927 00:33:51.860678 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:51.863447 13377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:51.868595 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13309
I20260927 00:33:51.868698 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:51.868973 10287 external_mini_cluster.cc:1428] Setting key 33f68c36edb0655931b51a89307514c3
I20260927 00:33:51.869105 13377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: Bootstrap starting.
I20260927 00:33:51.871721 13377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:51.872485 13377 log.cc:824] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:51.874696 13377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: No bootstrap required, opened a new log
Sep 27 00:33:51 dist-test-slave-6gxc krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:51.879415 13377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } }
I20260927 00:33:51.879714 13377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:51.879807 13377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339766ca302e4576b46d68da9949aab6, State: Initialized, Role: FOLLOWER
I20260927 00:33:51.880357 13377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } }
I20260927 00:33:51.880528 13377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:51.880628 13377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:51.880738 13377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:51.881875 13377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } }
I20260927 00:33:51.882261 13377 leader_election.cc:302] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 339766ca302e4576b46d68da9949aab6; no voters:
I20260927 00:33:51.882627 13377 leader_election.cc:288] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:51.882688 13382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:51.882948 13382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [term 1 LEADER]: Becoming Leader. State: Replica: 339766ca302e4576b46d68da9949aab6, State: Running, Role: LEADER
I20260927 00:33:51.883453 13382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } }
I20260927 00:33:51.883998 13377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:51.884133 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:51.870317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.10.11.254:37405)
0927 00:33:51.870704 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:51.870719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:51.870752 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:33:51.871328 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:51.871353 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:51.871653 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:51.871985 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:51.872766 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:51.874320 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:51.875261 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:51.875986 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:51.876158 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:51.879051 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:51.879056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:51.879068 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:51.879093 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:51.881103 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:51.881614 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:51.881618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:51.881622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:51.881725 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:51.882081 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:51.882084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:51.882086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:51.882574 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:51.882768 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:51.883074 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 00:33:51.883335 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":122,"threads_started":1}
I20260927 00:33:51.884681 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "339766ca302e4576b46d68da9949aab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } } }
I20260927 00:33:51.884907 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:51.886395 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339766ca302e4576b46d68da9949aab6. Latest consensus state: current_term: 1 leader_uuid: "339766ca302e4576b46d68da9949aab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339766ca302e4576b46d68da9949aab6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 37405 } } }
I20260927 00:33:51.887072 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6 [sys.catalog]: This master's current role is: LEADER
W20260927 00:33:51.888953 13398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:51.889037 13398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:51.889607 13391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:51.890492 13391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:51.895939 13391 catalog_manager.cc:1347] Generated new cluster ID: 9d6bd9f24dff4b2b9335805847da875a
I20260927 00:33:51.896030 13391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:51.901065 13391 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:51.901945 13391 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:51.908275 13391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 339766ca302e4576b46d68da9949aab6: Generated new TSK 0
I20260927 00:33:51.909101 13391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:51.936601 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:52.044106 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:52.044373 13403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:52.044452 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:52.048686 13403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:52.048790 13403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:52.048887 13403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:52.053905 13403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:52.054822 13403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:52.055713 13403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:52.064467 13411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:52.064481 13409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:52.064549 13403 server_base.cc:1034] running on GCE node
W20260927 00:33:52.064725 13408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:52.065076 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:52.065609 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:52.066916 13403 hybrid_clock.cc:648] HybridClock initialized: now 1790469232066904 us; error 25 us; skew 500 ppm
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:52.069540 13403 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:52.070384 13403 webserver.cc:460] Webserver started at http://127.10.11.193:42693/ using document root <none> and password file <none>
I20260927 00:33:52.070851 13403 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:52.070905 13403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:52.071050 13403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:52.072979 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "bc3af5200a914922b0826788133fb7bf"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "455bd9f1d93869aafba43b07b38a1731"
server_key_iv: "e79b71c51684139b0dce974ad2dd1813"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.073601 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "bc3af5200a914922b0826788133fb7bf"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "455bd9f1d93869aafba43b07b38a1731"
server_key_iv: "e79b71c51684139b0dce974ad2dd1813"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.077867 13403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:33:52.080205 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.081485 13403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:52.081620 13403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "bc3af5200a914922b0826788133fb7bf"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "455bd9f1d93869aafba43b07b38a1731"
server_key_iv: "e79b71c51684139b0dce974ad2dd1813"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.081758 13403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:52.119380 13403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:52.122545 13403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:52.122725 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:52.123324 13403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:52.124140 13403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:52.124187 13403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.124228 13403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:52.124243 13403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.132616 13403 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:37329
I20260927 00:33:52.132682 13531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:37329 every 8 connection(s)
I20260927 00:33:52.133450 13403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:52.142794 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13403
I20260927 00:33:52.142944 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:52.143271 10287 external_mini_cluster.cc:1428] Setting key 6f71f3dbf3124380d18e112d99a03d1b
I20260927 00:33:52.147409 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.135128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:57635 (local address 127.10.11.254:37405)
0927 00:33:52.135270 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:52.135273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:52.136054 (+ 781us) server_negotiation.cc:407] Connection header received
0927 00:33:52.137416 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:52.137420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:52.137479 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:52.137596 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:52.138762 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.139309 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.140454 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.140691 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.140820 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.143313 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:52.143319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:52.143322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:52.143354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:52.145239 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.145703 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.145707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.145709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.145813 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.146143 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.146149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.146154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.146436 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:52.146559 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:52.147182 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 00:33:52.147295 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:33:52.148226 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.135477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:37405 (local address 127.10.11.193:57635)
0927 00:33:52.135902 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:52.135941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:52.137205 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:52.137763 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:52.137770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:52.138149 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:52.138599 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.138612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.139445 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.139449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.140318 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.140328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.140849 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.140854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.141028 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.141568 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:52.141584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:52.143151 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:52.145353 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.145357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.145367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.145598 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.145913 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.145915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.145916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.146031 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.146572 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:52.146581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:52.146918 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 00:33:52.147128 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 00:33:52.147459 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260927 00:33:52.149423 13532 heartbeater.cc:347] Connected to a master server at 127.10.11.254:37405
I20260927 00:33:52.149658 13532 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:52.150292 13532 heartbeater.cc:511] Master 127.10.11.254:37405 requested a full tablet report, sending...
I20260927 00:33:52.151918 13341 ts_manager.cc:194] Registered new tserver with Master: bc3af5200a914922b0826788133fb7bf (127.10.11.193:37329)
I20260927 00:33:52.152971 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:57635
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:52.168983 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:52.273975 13537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:52.274242 13537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:52.274284 13537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:52.278163 13537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:52.278218 13537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:52.278298 13537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:52.283236 13537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:52.284186 13537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:52.285198 13537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:33:52.294056 13537 server_base.cc:1034] running on GCE node
W20260927 00:33:52.294056 13543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:52.294162 13542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:52.294162 13545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:52.294677 13537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:52.295257 13537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:52.296439 13537 hybrid_clock.cc:648] HybridClock initialized: now 1790469232296410 us; error 41 us; skew 500 ppm
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:52.299088 13537 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:52.299896 13537 webserver.cc:460] Webserver started at http://127.10.11.194:36523/ using document root <none> and password file <none>
I20260927 00:33:52.300448 13537 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:52.300534 13537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:52.300748 13537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:52.302611 13537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "65042115d30a432eaefbafb5f204fce7"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "19f9f0a88180c4efdd3f31befd557a41"
server_key_iv: "01ce17e77ab9eb7a9dea40216fb06a11"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.303138 13537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "65042115d30a432eaefbafb5f204fce7"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "19f9f0a88180c4efdd3f31befd557a41"
server_key_iv: "01ce17e77ab9eb7a9dea40216fb06a11"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.306499 13537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:52.308669 13552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.309986 13537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:52.310133 13537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "65042115d30a432eaefbafb5f204fce7"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "19f9f0a88180c4efdd3f31befd557a41"
server_key_iv: "01ce17e77ab9eb7a9dea40216fb06a11"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.310247 13537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:52.338876 13537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:52.341630 13537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:52.341837 13537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:52.342418 13537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:52.343197 13537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:52.343267 13537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.343335 13537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:52.343390 13537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.351747 13537 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:45329
I20260927 00:33:52.351771 13665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:45329 every 8 connection(s)
I20260927 00:33:52.352569 13537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:52.354421 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13537
I20260927 00:33:52.354560 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:52.354836 10287 external_mini_cluster.cc:1428] Setting key 33d3da82abaaeec5f7151b94d77f506b
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:33:52.367333 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.354297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40683 (local address 127.10.11.254:37405)
0927 00:33:52.354441 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:52.354446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:52.355345 (+ 899us) server_negotiation.cc:407] Connection header received
0927 00:33:52.356751 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:52.356758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:52.356832 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:52.356918 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:52.358770 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.359296 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.360449 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.360685 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.360797 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.363402 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:52.363405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:52.363407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:52.363428 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:52.364941 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.365509 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.365515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.365519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.365616 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.365947 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.365951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.365953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.366181 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:52.366347 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:52.366819 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 00:33:52.367146 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:33:52.368084 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.354756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:37405 (local address 127.10.11.194:40683)
0927 00:33:52.355200 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:52.355250 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:52.356503 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:52.357229 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:52.357240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:52.357919 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:52.358612 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.358633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.359404 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.359407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.360322 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.360333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.360841 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.360846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.361001 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.361592 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:52.361607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:52.363216 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:52.365060 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.365067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.365079 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.365326 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.365709 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.365714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.365716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.365848 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.366345 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:52.366350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:52.366554 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 00:33:52.366699 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:33:52.367058 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260927 00:33:52.369351 13666 heartbeater.cc:347] Connected to a master server at 127.10.11.254:37405
I20260927 00:33:52.369589 13666 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:52.370050 13666 heartbeater.cc:511] Master 127.10.11.254:37405 requested a full tablet report, sending...
I20260927 00:33:52.371152 13341 ts_manager.cc:194] Registered new tserver with Master: 65042115d30a432eaefbafb5f204fce7 (127.10.11.194:45329)
I20260927 00:33:52.371713 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:40683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:52.383327 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:52.488937 13671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:52.489182 13671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:52.489249 13671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:52.493181 13671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:52.493258 13671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:52.493345 13671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:52.498260 13671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:52.499154 13671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:52.500042 13671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:52.509053 13677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:52.509047 13676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:52.509037 13679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:52.509096 13671 server_base.cc:1034] running on GCE node
I20260927 00:33:52.509613 13671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:52.510290 13671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:52.511458 13671 hybrid_clock.cc:648] HybridClock initialized: now 1790469232511442 us; error 35 us; skew 500 ppm
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:52.514259 13671 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:52.515046 13671 webserver.cc:460] Webserver started at http://127.10.11.195:42081/ using document root <none> and password file <none>
I20260927 00:33:52.515504 13671 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:52.515558 13671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:52.515697 13671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:52.517441 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "0a99967a71c74e0a9fc32ce419fb2227"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "088d0d570309cf187ef6a083e57f8025"
server_key_iv: "bc2647712fb8fbe0ed124d751b95c762"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.517910 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "0a99967a71c74e0a9fc32ce419fb2227"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "088d0d570309cf187ef6a083e57f8025"
server_key_iv: "bc2647712fb8fbe0ed124d751b95c762"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.521303 13671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:33:52.523586 13686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.524710 13671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:52.524829 13671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "0a99967a71c74e0a9fc32ce419fb2227"
format_stamp: "Formatted at 2026-09-27 00:33:52 on dist-test-slave-6gxc"
server_key: "088d0d570309cf187ef6a083e57f8025"
server_key_iv: "bc2647712fb8fbe0ed124d751b95c762"
server_key_version: "encryptionkey@0"
I20260927 00:33:52.524924 13671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:52.536868 13671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:52.539742 13671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:52.539942 13671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:52.540438 13671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:52.541136 13671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:52.541179 13671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.541239 13671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:52.541282 13671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:52.549754 13671 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:43977
I20260927 00:33:52.549782 13799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:43977 every 8 connection(s)
I20260927 00:33:52.550545 13671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:52.559288 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13671
I20260927 00:33:52.559419 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:52.559804 10287 external_mini_cluster.cc:1428] Setting key 22a7277d2923e53254dc8aa9cf55aa0f
I20260927 00:33:52.563014 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.552092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:41203 (local address 127.10.11.254:37405)
0927 00:33:52.552215 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:52.552218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:52.552958 (+ 740us) server_negotiation.cc:407] Connection header received
0927 00:33:52.554235 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:52.554239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:52.554291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:52.554373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:52.555375 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.555903 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.556985 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.557214 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.557328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.559563 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:52.559567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:52.559569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:52.559601 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:52.561068 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.561541 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.561546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.561548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.561625 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.561950 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.561955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.561957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.562189 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:52.562288 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:52.562736 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 00:33:52.562855 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:33:52.563665 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.552401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:37405 (local address 127.10.11.195:41203)
0927 00:33:52.552828 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:52.552864 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:52.554022 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:52.554519 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:52.554525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:52.554832 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:52.555235 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.555246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.556025 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.556029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.556862 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:52.556870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.557355 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.557358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:52.557517 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.557950 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:52.557964 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:52.559363 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:52.561178 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.561183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.561195 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.561440 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.561723 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:52.561726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:52.561728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:52.561858 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:52.562297 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:52.562305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:52.562519 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:33:52.562697 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:33:52.562976 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260927 00:33:52.564648 13800 heartbeater.cc:347] Connected to a master server at 127.10.11.254:37405
I20260927 00:33:52.564833 13800 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:52.565213 13800 heartbeater.cc:511] Master 127.10.11.254:37405 requested a full tablet report, sending...
I20260927 00:33:52.566151 13341 ts_manager.cc:194] Registered new tserver with Master: 0a99967a71c74e0a9fc32ce419fb2227 (127.10.11.195:43977)
I20260927 00:33:52.566735 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:41203
I20260927 00:33:52.573622 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469232, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:52.598308 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:52.589758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.10.11.254:37405)
0927 00:33:52.589882 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:52.589885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:52.590069 (+ 184us) server_negotiation.cc:407] Connection header received
0927 00:33:52.590241 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:52.590244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:52.590282 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:52.590356 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:52.591077 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.591568 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.592472 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:52.592676 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:52.592763 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:52.595173 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:52.595175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:52.595177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:52.595198 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:52.596608 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.597014 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.597016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.597018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.597095 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:52.597379 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:52.597382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:52.597383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:52.597591 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:52.597699 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:52.598021 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 00:33:52.598197 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:33:52.598649 13341 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59046
I20260927 00:33:52.599514 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13403
I20260927 00:33:52.605020 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13537
I20260927 00:33:52.610383 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13671
I20260927 00:33:52.615666 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13309
2026-09-27T00:33:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2096 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13812](info): set up 2 sockets
Sep 27 00:33:52 dist-test-slave-6gxc krb5kdc[13812](info): commencing operation
krb5kdc: starting...
W20260927 00:33:53.642443 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:53 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:53.755203 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39443
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:53.859510 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:53.859735 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:53.859781 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:53.863581 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:53.863629 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:53.863648 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:53.863664 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:53.863679 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:53.868608 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:53.869460 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:53.870365 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:53.878430 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:53.878580 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:53.878575 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:53.878722 13825 server_base.cc:1034] running on GCE node
I20260927 00:33:53.879280 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:53.880311 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:53.881505 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790469233881469 us; error 51 us; skew 500 ppm
Sep 27 00:33:53 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469233, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:53.888451 13825 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:53.889216 13825 webserver.cc:460] Webserver started at http://127.10.11.254:33821/ using document root <none> and password file <none>
I20260927 00:33:53.889709 13825 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:53.889765 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:53.889917 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:53.891865 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "0aed84230dd047999691df6eb3b331c4"
format_stamp: "Formatted at 2026-09-27 00:33:53 on dist-test-slave-6gxc"
server_key: "2f5150ca84f3331d6ed802d2ef30e434"
server_key_iv: "6cd90803755158a37cacdb6cba781cc9"
server_key_version: "encryptionkey@0"
I20260927 00:33:53.892347 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "0aed84230dd047999691df6eb3b331c4"
format_stamp: "Formatted at 2026-09-27 00:33:53 on dist-test-slave-6gxc"
server_key: "2f5150ca84f3331d6ed802d2ef30e434"
server_key_iv: "6cd90803755158a37cacdb6cba781cc9"
server_key_version: "encryptionkey@0"
I20260927 00:33:53.895758 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:53.897884 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:53.899186 13825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:33:53.899323 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "0aed84230dd047999691df6eb3b331c4"
format_stamp: "Formatted at 2026-09-27 00:33:53 on dist-test-slave-6gxc"
server_key: "2f5150ca84f3331d6ed802d2ef30e434"
server_key_iv: "6cd90803755158a37cacdb6cba781cc9"
server_key_version: "encryptionkey@0"
I20260927 00:33:53.899427 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:53.919919 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:53.922902 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:53.923089 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:53.929677 13825 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:39443
I20260927 00:33:53.929704 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:39443 every 8 connection(s)
I20260927 00:33:53.930593 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:53.933103 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:53.938313 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: Bootstrap starting.
I20260927 00:33:53.940418 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13825
I20260927 00:33:53.940555 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:53.940871 10287 external_mini_cluster.cc:1428] Setting key 057b7ae0aed9193744f228f8c51ace1e
I20260927 00:33:53.941269 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:53.942283 13893 log.cc:824] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:53.944680 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: No bootstrap required, opened a new log
I20260927 00:33:53.949536 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } }
I20260927 00:33:53.949829 13893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:53.949927 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aed84230dd047999691df6eb3b331c4, State: Initialized, Role: FOLLOWER
I20260927 00:33:53.950465 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } }
I20260927 00:33:53.950640 13893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:53.950709 13893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:53.950802 13893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:53.951678 13893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } }
Sep 27 00:33:53 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:53.952036 13893 leader_election.cc:302] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aed84230dd047999691df6eb3b331c4; no voters:
I20260927 00:33:53.952353 13893 leader_election.cc:288] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:53.952461 13898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:53.952664 13898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [term 1 LEADER]: Becoming Leader. State: Replica: 0aed84230dd047999691df6eb3b331c4, State: Running, Role: LEADER
I20260927 00:33:53.952996 13898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } }
I20260927 00:33:53.953647 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:53.954427 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aed84230dd047999691df6eb3b331c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } } }
I20260927 00:33:53.954571 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:53.954891 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aed84230dd047999691df6eb3b331c4. Latest consensus state: current_term: 1 leader_uuid: "0aed84230dd047999691df6eb3b331c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aed84230dd047999691df6eb3b331c4" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39443 } } }
I20260927 00:33:53.954991 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:53.957969 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:53.943340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.10.11.254:39443)
0927 00:33:53.943762 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:53.943784 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:53.943830 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:33:53.944602 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:53.944619 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:53.944986 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:53.945316 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:53.946172 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:53.947802 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:53.948971 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:53.949642 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:53.949811 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:53.952730 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:53.952735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:53.952750 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:53.952778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:53.954950 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:53.955500 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:53.955507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:53.955514 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:53.955637 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:53.955950 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:53.955954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:53.955955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:53.956427 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:53.956623 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:53.956842 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:33:53.957119 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
W20260927 00:33:53.959070 13914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:53.959129 13914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:53.959213 13907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:53.960007 13907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:53.965440 13907 catalog_manager.cc:1347] Generated new cluster ID: 09f59ca9c37e4d4f98853406292e5399
I20260927 00:33:53.965523 13907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:53.975520 13907 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:53.976693 13907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:53.985245 13907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aed84230dd047999691df6eb3b331c4: Generated new TSK 0
I20260927 00:33:53.985931 13907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:54.019201 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:54.125789 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:54.126040 13919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:54.126129 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:54.130057 13919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:54.130151 13919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:54.130265 13919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:54.135167 13919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:54.136076 13919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:54.136981 13919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:54.145746 13924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.145781 13919 server_base.cc:1034] running on GCE node
W20260927 00:33:54.145757 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:54.145797 13927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.146473 13919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:54.147030 13919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:54.148208 13919 hybrid_clock.cc:648] HybridClock initialized: now 1790469234148189 us; error 29 us; skew 500 ppm
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:54.150825 13919 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:54.151567 13919 webserver.cc:460] Webserver started at http://127.10.11.193:37237/ using document root <none> and password file <none>
I20260927 00:33:54.152074 13919 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:54.152161 13919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:54.152375 13919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:54.154402 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "fa58dd3d5cc542db9b4c1bb54a193a9f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "b63e25c6e4917e7bb2dd97b36f0f46b0"
server_key_iv: "226bb872606334d7e5f77d7352fe52e0"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.154969 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "fa58dd3d5cc542db9b4c1bb54a193a9f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "b63e25c6e4917e7bb2dd97b36f0f46b0"
server_key_iv: "226bb872606334d7e5f77d7352fe52e0"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.158363 13919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:54.160552 13934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.161682 13919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:54.161844 13919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "fa58dd3d5cc542db9b4c1bb54a193a9f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "b63e25c6e4917e7bb2dd97b36f0f46b0"
server_key_iv: "226bb872606334d7e5f77d7352fe52e0"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.161945 13919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:54.176761 13919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:54.179682 13919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:54.179875 13919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:54.180392 13919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:54.181108 13919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:54.181176 13919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.181238 13919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:54.181296 13919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.189865 13919 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44869
I20260927 00:33:54.189904 14047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44869 every 8 connection(s)
I20260927 00:33:54.190650 13919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:54.194622 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 13919
I20260927 00:33:54.194757 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:54.195041 10287 external_mini_cluster.cc:1428] Setting key 9c140feccebb545198f7bd9945256c9a
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:33:54.203588 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.192231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:35347 (local address 127.10.11.254:39443)
0927 00:33:54.192350 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:54.192354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:54.193111 (+ 757us) server_negotiation.cc:407] Connection header received
0927 00:33:54.194456 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:54.194460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:54.194519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:54.194621 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:54.195782 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.196324 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.197400 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.197648 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.197757 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.200081 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:54.200084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:54.200086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:54.200107 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:54.201550 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.202048 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.202053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.202055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.202189 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.202507 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.202512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.202514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.202683 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:54.202795 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:54.203302 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 00:33:54.203424 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:33:54.204319 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.192571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39443 (local address 127.10.11.193:35347)
0927 00:33:54.192974 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:54.193013 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:54.194234 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:54.194834 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:54.194843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:54.195208 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:54.195631 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.195643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.196444 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.196447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.197277 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.197285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.197794 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.197798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.197953 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.198548 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:54.198563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:54.199925 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:54.201686 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.201692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.201704 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.201946 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.202280 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.202283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.202285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.202414 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.202783 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:54.202789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:54.203063 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 00:33:54.203258 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:33:54.203544 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260927 00:33:54.205439 14048 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39443
I20260927 00:33:54.205623 14048 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:33:54.206151 14048 heartbeater.cc:511] Master 127.10.11.254:39443 requested a full tablet report, sending...
I20260927 00:33:54.207525 13857 ts_manager.cc:194] Registered new tserver with Master: fa58dd3d5cc542db9b4c1bb54a193a9f (127.10.11.193:44869)
I20260927 00:33:54.208565 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:35347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:54.219338 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:54.324404 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:54.324630 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:54.324668 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:54.328624 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:54.328678 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:54.328764 14053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:54.333842 14053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:54.334862 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:54.335839 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:54.344879 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:54.344888 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.345166 14053 server_base.cc:1034] running on GCE node
W20260927 00:33:54.344889 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.345568 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:54.346159 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:54.347323 14053 hybrid_clock.cc:648] HybridClock initialized: now 1790469234347311 us; error 25 us; skew 500 ppm
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:54.349843 14053 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:54.350649 14053 webserver.cc:460] Webserver started at http://127.10.11.194:46073/ using document root <none> and password file <none>
I20260927 00:33:54.351150 14053 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:54.351244 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:54.351450 14053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:54.353307 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "cf2b5f55ef874b83a2829a4346fdc35e"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "4ad9871e7fdafdb51cf03b636ea55ed0"
server_key_iv: "d23f6760908a70ad64cc524b5a97939d"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.353847 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "cf2b5f55ef874b83a2829a4346fdc35e"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "4ad9871e7fdafdb51cf03b636ea55ed0"
server_key_iv: "d23f6760908a70ad64cc524b5a97939d"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.357514 14053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:33:54.359869 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.361155 14053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:54.361290 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "cf2b5f55ef874b83a2829a4346fdc35e"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "4ad9871e7fdafdb51cf03b636ea55ed0"
server_key_iv: "d23f6760908a70ad64cc524b5a97939d"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.361392 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:54.371881 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:54.374505 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:54.374679 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:54.375180 14053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:54.375916 14053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:54.375983 14053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.376044 14053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:54.376086 14053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.384047 14053 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:33149
I20260927 00:33:54.384078 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:33149 every 8 connection(s)
I20260927 00:33:54.384837 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:54.394454 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14053
I20260927 00:33:54.394575 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:54.394858 10287 external_mini_cluster.cc:1428] Setting key 60f3ad3455f0d79f36da1149448f74fa
I20260927 00:33:54.398294 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.386657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54209 (local address 127.10.11.254:39443)
0927 00:33:54.386781 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:54.386785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:54.387537 (+ 752us) server_negotiation.cc:407] Connection header received
0927 00:33:54.388809 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:54.388812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:54.388855 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:54.388930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:54.390013 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.390604 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.391688 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.391936 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.392020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.394406 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:54.394408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:54.394410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:54.394433 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:54.396190 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.396786 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.396789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.396790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.396864 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.397266 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.397268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.397270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.397440 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:54.397562 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:54.397994 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 00:33:54.398131 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:33:54.399091 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.386994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39443 (local address 127.10.11.194:54209)
0927 00:33:54.387381 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:54.387418 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:54.388611 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:54.389115 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:54.389122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:54.389453 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:54.389874 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.389886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.390711 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.390714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.391564 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.391573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.392061 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.392063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.392189 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.392764 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:54.392778 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:54.394238 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:54.396293 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.396297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.396307 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.396644 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.396963 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.396969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.396974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.397149 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.397562 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:54.397569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:54.397775 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 00:33:54.397954 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:33:54.398269 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
I20260927 00:33:54.400133 14182 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39443
I20260927 00:33:54.400341 14182 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:54.400763 14182 heartbeater.cc:511] Master 127.10.11.254:39443 requested a full tablet report, sending...
I20260927 00:33:54.401736 13857 ts_manager.cc:194] Registered new tserver with Master: cf2b5f55ef874b83a2829a4346fdc35e (127.10.11.194:33149)
I20260927 00:33:54.402572 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:54209
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:54.420593 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:54.525561 14187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:54.525779 14187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:54.525817 14187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:54.529682 14187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:54.529731 14187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:54.529790 14187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:54.534615 14187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:54.535465 14187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:54.536350 14187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:54.545203 14193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:54.545291 14195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.545404 14187 server_base.cc:1034] running on GCE node
W20260927 00:33:54.545291 14192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:54.545955 14187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:54.546566 14187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:54.547752 14187 hybrid_clock.cc:648] HybridClock initialized: now 1790469234547739 us; error 25 us; skew 500 ppm
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:54.550329 14187 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:54.551084 14187 webserver.cc:460] Webserver started at http://127.10.11.195:34913/ using document root <none> and password file <none>
I20260927 00:33:54.551592 14187 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:54.551676 14187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:54.551904 14187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:54.553851 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "a21fea16b785452ea6c18f013abcd86f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "e75ff94fe79e94fb118c08a19b393d12"
server_key_iv: "84f7777211eafb9a8b24de06113dd6a7"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.554415 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "a21fea16b785452ea6c18f013abcd86f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "e75ff94fe79e94fb118c08a19b393d12"
server_key_iv: "84f7777211eafb9a8b24de06113dd6a7"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.557940 14187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:33:54.560248 14202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.561385 14187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:54.561517 14187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "a21fea16b785452ea6c18f013abcd86f"
format_stamp: "Formatted at 2026-09-27 00:33:54 on dist-test-slave-6gxc"
server_key: "e75ff94fe79e94fb118c08a19b393d12"
server_key_iv: "84f7777211eafb9a8b24de06113dd6a7"
server_key_version: "encryptionkey@0"
I20260927 00:33:54.561619 14187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:54.572729 14187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:54.575492 14187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:54.575670 14187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:54.576177 14187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:54.576921 14187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:54.576989 14187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.577054 14187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:54.577102 14187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:54.585752 14187 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:42453
I20260927 00:33:54.585789 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:42453 every 8 connection(s)
I20260927 00:33:54.586597 14187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:54.595664 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14187
I20260927 00:33:54.595814 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:54.596099 10287 external_mini_cluster.cc:1428] Setting key cd75d365cdb4bed13ba6228bb1131738
I20260927 00:33:54.599447 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.588254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:40129 (local address 127.10.11.254:39443)
0927 00:33:54.588425 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:54.588430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:54.589210 (+ 780us) server_negotiation.cc:407] Connection header received
0927 00:33:54.590552 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:54.590555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:54.590601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:54.590677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:54.591751 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.592274 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.593387 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.593598 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.593681 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.595951 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:54.595959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:54.595961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:54.595986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:54.597615 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.598085 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.598088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.598090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.598199 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.598560 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.598562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.598563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.598727 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:54.598807 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:54.599237 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 00:33:54.599337 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:33:54.600138 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.588656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39443 (local address 127.10.11.195:40129)
0927 00:33:54.589065 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:54.589102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:54.590334 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:54.590831 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:54.590837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:54.591150 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:54.591607 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.591620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.592392 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.592395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.593254 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:54.593270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.593724 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.593726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:54.593861 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.594412 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:54.594427 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:54.595787 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:54.597722 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.597726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.597737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.597986 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.598288 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:54.598290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:54.598292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:54.598476 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:54.598813 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:54.598818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:54.599027 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 00:33:54.599172 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:33:54.599403 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260927 00:33:54.601078 14316 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39443
I20260927 00:33:54.601297 14316 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:54.601742 14316 heartbeater.cc:511] Master 127.10.11.254:39443 requested a full tablet report, sending...
I20260927 00:33:54.602735 13857 ts_manager.cc:194] Registered new tserver with Master: a21fea16b785452ea6c18f013abcd86f (127.10.11.195:42453)
I20260927 00:33:54.603364 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:40129
I20260927 00:33:54.609836 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:33:54.634218 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.625317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.10.11.193:44869)
0927 00:33:54.625647 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:54.625657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:54.625680 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:33:54.625725 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:54.625730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:54.625935 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:54.626051 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:54.626728 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.627513 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.628322 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.628676 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.628769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.631079 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:54.631087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:54.631098 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:54.631120 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:54.632528 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.632924 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.632929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.632930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.633010 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.633244 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.633247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.633249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.633454 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:54.633597 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:54.633813 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 00:33:54.633986 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:54.644445 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:54.636741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.10.11.254:39443)
0927 00:33:54.636895 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:54.636902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:54.636961 (+ 59us) server_negotiation.cc:407] Connection header received
0927 00:33:54.637087 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:54.637089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:54.637125 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:54.637187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:54.637717 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.638294 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.639017 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:54.639222 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:54.639308 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:54.641568 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:54.641571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:54.641573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:54.641596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:54.643062 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.643435 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.643438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.643439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.643519 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:54.643764 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:54.643766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:54.643767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:54.643962 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:54.644060 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:54.644229 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:33:54.644328 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:33:54.644843 13857 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48058
I20260927 00:33:54.645408 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13919
I20260927 00:33:54.650712 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14053
I20260927 00:33:54.656040 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14187
I20260927 00:33:54.661266 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 13825
2026-09-27T00:33:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2045 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[14328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[14328](info): set up 2 sockets
Sep 27 00:33:54 dist-test-slave-6gxc krb5kdc[14328](info): commencing operation
krb5kdc: starting...
W20260927 00:33:55.698559 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:33:55 dist-test-slave-6gxc krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:33:55.811193 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34653
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:34653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:55.915714 14341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:55.915982 14341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:55.916054 14341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:55.920115 14341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:55.920193 14341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:55.920228 14341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:55.920266 14341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:55.920297 14341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:55.925222 14341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:34653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:55.926209 14341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:55.927127 14341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:55.934885 14347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:55.934855 14346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:55.934890 14349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:55.935274 14341 server_base.cc:1034] running on GCE node
I20260927 00:33:55.935766 14341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:55.936764 14341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:55.937974 14341 hybrid_clock.cc:648] HybridClock initialized: now 1790469235937945 us; error 40 us; skew 500 ppm
Sep 27 00:33:55 dist-test-slave-6gxc krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:55.940743 14341 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:33:55.941380 14341 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:55.941438 14341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:55.941591 14341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:55.943507 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "b75ce0191566467da5cc91ab36d878ac"
format_stamp: "Formatted at 2026-09-27 00:33:55 on dist-test-slave-6gxc"
server_key: "652c8e3aa50a21268d0809bdd0218012"
server_key_iv: "2f735b3f823f722bc33ccbbc05c9f53b"
server_key_version: "encryptionkey@0"
I20260927 00:33:55.943981 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "b75ce0191566467da5cc91ab36d878ac"
format_stamp: "Formatted at 2026-09-27 00:33:55 on dist-test-slave-6gxc"
server_key: "652c8e3aa50a21268d0809bdd0218012"
server_key_iv: "2f735b3f823f722bc33ccbbc05c9f53b"
server_key_version: "encryptionkey@0"
I20260927 00:33:55.947535 14341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:55.949815 14355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:55.951040 14341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:55.951138 14341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "b75ce0191566467da5cc91ab36d878ac"
format_stamp: "Formatted at 2026-09-27 00:33:55 on dist-test-slave-6gxc"
server_key: "652c8e3aa50a21268d0809bdd0218012"
server_key_iv: "2f735b3f823f722bc33ccbbc05c9f53b"
server_key_version: "encryptionkey@0"
I20260927 00:33:55.951218 14341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:55.966084 14341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:55.969135 14341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:55.969272 14341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:55.976267 14341 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:34653
I20260927 00:33:55.976526 14407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:34653 every 8 connection(s)
I20260927 00:33:55.976873 14341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:55.979457 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:55.984524 14408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: Bootstrap starting.
I20260927 00:33:55.986179 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14341
I20260927 00:33:55.986317 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:55.986572 10287 external_mini_cluster.cc:1428] Setting key 4f06a4108f200b0ca7222397fa0baa38
I20260927 00:33:55.986934 14408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:55.987789 14408 log.cc:824] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:55.989679 14408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: No bootstrap required, opened a new log
I20260927 00:33:55.993692 14408 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } }
I20260927 00:33:55.993960 14408 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:55.994026 14408 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75ce0191566467da5cc91ab36d878ac, State: Initialized, Role: FOLLOWER
I20260927 00:33:55.994534 14408 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } }
I20260927 00:33:55.994716 14408 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:55.994804 14408 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:55.994899 14408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:33:55 dist-test-slave-6gxc krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:55.995800 14408 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } }
I20260927 00:33:55.996160 14408 leader_election.cc:302] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75ce0191566467da5cc91ab36d878ac; no voters:
I20260927 00:33:55.996507 14408 leader_election.cc:288] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:55.996572 14413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:55.996760 14413 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [term 1 LEADER]: Becoming Leader. State: Replica: b75ce0191566467da5cc91ab36d878ac, State: Running, Role: LEADER
I20260927 00:33:55.997045 14413 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } }
I20260927 00:33:55.997668 14408 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:55.998172 14414 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75ce0191566467da5cc91ab36d878ac. Latest consensus state: current_term: 1 leader_uuid: "b75ce0191566467da5cc91ab36d878ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } } }
I20260927 00:33:55.998179 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75ce0191566467da5cc91ab36d878ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75ce0191566467da5cc91ab36d878ac" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34653 } } }
I20260927 00:33:55.998312 14414 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:55.998325 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:56.001412 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:55.987598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.10.11.254:34653)
0927 00:33:55.987982 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:55.988002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:55.988043 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:33:55.988791 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:55.988810 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:55.989109 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:55.989377 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:55.990282 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:55.991206 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:55.992395 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:55.992824 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:55.992933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:55.995745 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:55.995750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:55.995761 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:55.995786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:55.997558 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:55.998152 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:55.998158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:55.998165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:55.998408 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:55.998796 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:55.998801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:55.998804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:55.999270 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:55.999471 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.000265 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.000542 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":157,"threads_started":1}
W20260927 00:33:56.002612 14428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:56.002712 14428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:56.003603 14419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:56.004261 14419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:56.008687 14419 catalog_manager.cc:1347] Generated new cluster ID: ca9c7a46c99f4e4c8410194604e49999
I20260927 00:33:56.008746 14419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:56.021564 14419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:56.022495 14419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:56.028028 14419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: Generated new TSK 0
I20260927 00:33:56.028621 14419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:33:56.055269 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:56.164984 14434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:56.165213 14434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:56.165256 14434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:56.169193 14434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:56.169242 14434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:56.169337 14434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:56.174413 14434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:56.175318 14434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:56.176244 14434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:56.184999 14439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:56.185164 14434 server_base.cc:1034] running on GCE node
W20260927 00:33:56.185217 14440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:56.185005 14442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:56.185614 14434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:56.186225 14434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:56.187491 14434 hybrid_clock.cc:648] HybridClock initialized: now 1790469236187480 us; error 25 us; skew 500 ppm
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:56.190301 14434 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:33:56.190928 14434 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:56.190987 14434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:56.191134 14434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:56.192943 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "04da11291e9f487d97e974f89bcf1143"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "8f429bafb61ea70529746e43dfb7aa62"
server_key_iv: "c0891a3a9307b2d3cdbd796f50f5f1bc"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.193428 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "04da11291e9f487d97e974f89bcf1143"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "8f429bafb61ea70529746e43dfb7aa62"
server_key_iv: "c0891a3a9307b2d3cdbd796f50f5f1bc"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.196823 14434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 00:33:56.199456 14448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.200685 14434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:56.200791 14434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "04da11291e9f487d97e974f89bcf1143"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "8f429bafb61ea70529746e43dfb7aa62"
server_key_iv: "c0891a3a9307b2d3cdbd796f50f5f1bc"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.200868 14434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:56.221992 14434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:56.225014 14434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:56.225203 14434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:56.225713 14434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:56.226475 14434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:56.226518 14434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.226579 14434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:56.226611 14434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.235491 14434 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:36221
I20260927 00:33:56.235517 14561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:36221 every 8 connection(s)
I20260927 00:33:56.236017 14434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:56.240608 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14434
I20260927 00:33:56.240747 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:56.241039 10287 external_mini_cluster.cc:1428] Setting key a568b1859c348d2f035e4469f59d8048
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:56.249197 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.237529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:49721 (local address 127.10.11.254:34653)
0927 00:33:56.237661 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.237664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.238481 (+ 817us) server_negotiation.cc:407] Connection header received
0927 00:33:56.239753 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.239756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.239829 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.239921 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:56.241072 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.241715 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.242871 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.243106 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.243226 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.245759 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:56.245762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:56.245764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:56.245784 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:56.247281 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.247789 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.247794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.247796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.247875 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.248195 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.248199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.248201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.248370 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:56.248474 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.248914 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.249035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:33:56.249868 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.237859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34653 (local address 127.10.11.193:49721)
0927 00:33:56.238345 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.238384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.239532 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.240082 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.240088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.240403 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:56.240935 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.240951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.241833 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.241836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.242743 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.242752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.243255 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.243259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.243420 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.243960 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:56.243974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:56.245581 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:56.247412 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.247417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.247429 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.247679 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.247966 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.247970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.247972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.248101 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.248471 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:56.248477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:56.248691 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.248862 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.249143 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":102,"threads_started":1}
I20260927 00:33:56.250999 14562 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34653
I20260927 00:33:56.251224 14562 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:56.251647 14562 heartbeater.cc:511] Master 127.10.11.254:34653 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:33:56.253120 14372 ts_manager.cc:194] Registered new tserver with Master: 04da11291e9f487d97e974f89bcf1143 (127.10.11.193:36221)
I20260927 00:33:56.254839 14372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:49721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:33:56.266405 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:56.371402 14567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:56.371654 14567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:56.371728 14567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:56.375677 14567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:56.375748 14567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:56.375836 14567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:56.380748 14567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:56.381832 14567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:56.382831 14567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:56.391584 14572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:56.391554 14573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:56.391613 14567 server_base.cc:1034] running on GCE node
W20260927 00:33:56.391563 14575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:56.392210 14567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:56.392755 14567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:56.393944 14567 hybrid_clock.cc:648] HybridClock initialized: now 1790469236393931 us; error 23 us; skew 500 ppm
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:56.396602 14567 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:33:56.397267 14567 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:56.397356 14567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:56.397599 14567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:56.399835 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "492d0fa602b1461aa4cd95579b741d23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "d4df43390c9b215c4c9f9ee10ec8b85c"
server_key_iv: "c9c434cffba3790540601e9f06b59054"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.400389 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "492d0fa602b1461aa4cd95579b741d23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "d4df43390c9b215c4c9f9ee10ec8b85c"
server_key_iv: "c9c434cffba3790540601e9f06b59054"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.403792 14567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:56.406075 14581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.407222 14567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:56.407347 14567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "492d0fa602b1461aa4cd95579b741d23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "d4df43390c9b215c4c9f9ee10ec8b85c"
server_key_iv: "c9c434cffba3790540601e9f06b59054"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.407445 14567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:56.422768 14567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:56.425668 14567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:56.425853 14567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:56.426407 14567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:56.427119 14567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:56.427186 14567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.427248 14567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:56.427299 14567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.435598 14567 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:38461
I20260927 00:33:56.435650 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:38461 every 8 connection(s)
I20260927 00:33:56.436173 14567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:56.441636 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14567
I20260927 00:33:56.441769 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:56.442054 10287 external_mini_cluster.cc:1428] Setting key fef5691326b10b7666b5b4cb24e29276
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:56.449225 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.437816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:42383 (local address 127.10.11.254:34653)
0927 00:33:56.437959 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.437964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.438885 (+ 921us) server_negotiation.cc:407] Connection header received
0927 00:33:56.440062 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.440065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.440109 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.440190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:56.441240 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.441812 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.442998 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.443228 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.443345 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.445575 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:56.445579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:56.445581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:56.445612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:56.447102 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.447694 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.447696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.447697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.447769 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.448152 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.448154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.448156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.448326 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:56.448399 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.449023 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.449113 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:33:56.450313 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.438275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34653 (local address 127.10.11.194:42383)
0927 00:33:56.438726 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.438771 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.439855 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.440378 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.440385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.440690 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:56.441111 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.441124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.441924 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.441928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.442870 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.442879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.443387 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.443392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.443543 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.444090 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:56.444104 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:56.445417 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:56.447211 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.447220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.447236 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.447593 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.447858 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.447865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.447870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.448049 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.448430 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:56.448438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:56.448750 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.448980 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.449303 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":134,"threads_started":1}
I20260927 00:33:56.451453 14695 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34653
I20260927 00:33:56.451669 14695 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:33:56.452064 14695 heartbeater.cc:511] Master 127.10.11.254:34653 requested a full tablet report, sending...
I20260927 00:33:56.453050 14372 ts_manager.cc:194] Registered new tserver with Master: 492d0fa602b1461aa4cd95579b741d23 (127.10.11.194:38461)
I20260927 00:33:56.453586 14372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:42383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:33:56.468490 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:33:56.573020 14700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:56.573230 14700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:56.573269 14700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:56.577039 14700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:56.577082 14700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:56.577136 14700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:56.581909 14700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:56.582758 14700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:56.583648 14700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:56.592357 14705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:56.592396 14706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:56.592381 14708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:56.592581 14700 server_base.cc:1034] running on GCE node
I20260927 00:33:56.592998 14700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:56.593542 14700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:56.594703 14700 hybrid_clock.cc:648] HybridClock initialized: now 1790469236594686 us; error 31 us; skew 500 ppm
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:33:56.597340 14700 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:33:56.598024 14700 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:56.598150 14700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:56.598380 14700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:56.600366 14700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "2238addbb8f74ab485146628d1f90a23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "5d1295038d1a2447f0527507790901f6"
server_key_iv: "69b0ebb3d536a5e3614f7dd9ced7c2ea"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.600893 14700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "2238addbb8f74ab485146628d1f90a23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "5d1295038d1a2447f0527507790901f6"
server_key_iv: "69b0ebb3d536a5e3614f7dd9ced7c2ea"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.604415 14700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:33:56.606755 14714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.607955 14700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:56.608089 14700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "2238addbb8f74ab485146628d1f90a23"
format_stamp: "Formatted at 2026-09-27 00:33:56 on dist-test-slave-6gxc"
server_key: "5d1295038d1a2447f0527507790901f6"
server_key_iv: "69b0ebb3d536a5e3614f7dd9ced7c2ea"
server_key_version: "encryptionkey@0"
I20260927 00:33:56.608192 14700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:56.628072 14700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:56.630810 14700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:56.631000 14700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:56.631536 14700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:56.632313 14700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:56.632382 14700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.632450 14700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:56.632499 14700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:56.640848 14700 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:35447
I20260927 00:33:56.640864 14827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:35447 every 8 connection(s)
I20260927 00:33:56.641381 14700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:56.643739 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14700
I20260927 00:33:56.643908 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:56.644243 10287 external_mini_cluster.cc:1428] Setting key 7738bf29a7300e6dda785f2d53232bdc
Sep 27 00:33:56 dist-test-slave-6gxc krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:33:56.654215 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.642856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34907 (local address 127.10.11.254:34653)
0927 00:33:56.643027 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.643032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.643880 (+ 848us) server_negotiation.cc:407] Connection header received
0927 00:33:56.645306 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.645308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.645358 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.645424 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:56.646761 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.647274 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.648347 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.648556 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.648634 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.650956 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:56.650959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:56.650961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:56.650982 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:56.652233 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.652721 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.652723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.652725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.652820 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.653174 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.653176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.653177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.653353 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:56.653447 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.653891 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.653979 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:33:56.654987 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.643238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34653 (local address 127.10.11.195:34907)
0927 00:33:56.643718 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.643775 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.645108 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.645571 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.645578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.646016 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:56.646636 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.646651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.647381 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.647383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.648231 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.648240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.648672 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.648675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.648801 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.649359 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:33:56.649374 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:56.650815 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:56.652359 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.652363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.652373 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.652614 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.652940 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:33:56.652943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:33:56.652944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:33:56.653083 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:33:56.653477 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:56.653483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:56.653693 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.653852 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.654178 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260927 00:33:56.656019 14828 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34653
I20260927 00:33:56.656246 14828 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:56.656666 14828 heartbeater.cc:511] Master 127.10.11.254:34653 requested a full tablet report, sending...
I20260927 00:33:56.657763 14372 ts_manager.cc:194] Registered new tserver with Master: 2238addbb8f74ab485146628d1f90a23 (127.10.11.195:35447)
I20260927 00:33:56.658308 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:33:56.658526 14372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:34907
I20260927 00:33:56.666939 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.660261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.10.11.254:34653)
0927 00:33:56.660383 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.660388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.660531 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:33:56.660616 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.660618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.660651 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.660713 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:56.661276 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.661749 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.662625 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.662829 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.662921 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.664048 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:56.664051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:56.664052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:33:56.664072 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:56.665510 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.665861 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.665863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.665864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.665931 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:33:56.666247 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:33:56.666251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:33:56.666253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:33:56.666460 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:56.666571 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.666746 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.666837 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:33:56.669399 14372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:56.672355 14372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:56.686399 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.680186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35447 (local address 127.0.0.1:42910)
0927 00:33:56.680534 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.680581 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.680694 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.681760 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.681765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.681788 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:56.681912 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.681918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683442 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.683447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.684823 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.684832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.685836 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.685842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.685957 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.685991 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.686117 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.686238 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":342,"thread_start_us":89,"threads_started":1}
I20260927 00:33:56.686549 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.680733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.10.11.195:35447)
0927 00:33:56.681221 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.681232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.681253 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:33:56.681373 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.681379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.681582 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.681767 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:56.682041 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683288 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.684953 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.685675 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.685788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.685971 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.686305 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.686399 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":165,"threads_started":1}
I20260927 00:33:56.686774 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.680395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:38461 (local address 127.0.0.1:52934)
0927 00:33:56.681409 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.681425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.681507 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.682443 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.682446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.682461 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:56.682535 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.682539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683571 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.683573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.685074 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.685083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.686078 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.686082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.686253 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.686277 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.686348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.686414 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":178,"thread_start_us":87,"threads_started":1}
I20260927 00:33:56.686918 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.681245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.10.11.193:36221)
0927 00:33:56.681629 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.681641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.681667 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:33:56.681740 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.681744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.681923 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.682093 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:56.682306 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683375 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.685218 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.685780 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.685882 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.686017 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.686646 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.686757 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":89,"threads_started":1}
I20260927 00:33:56.687024 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.680972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.10.11.194:38461)
0927 00:33:56.681785 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.681797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.681817 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:33:56.681908 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.681912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.682091 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.682260 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:56.682639 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683328 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.685200 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.685971 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.686067 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.686211 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.686546 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.686640 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":118,"threads_started":1}
I20260927 00:33:56.686566 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.680861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:36221 (local address 127.0.0.1:51804)
0927 00:33:56.681488 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.681498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.681565 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.682060 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.682063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.682079 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:56.682219 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.682225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.683526 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.683530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.685072 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.685083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.686079 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.686085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.686253 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.686282 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.686351 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.686419 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":167,"thread_start_us":75,"threads_started":1}
I20260927 00:33:56.689777 14629 tablet_service.cc:1467] Processing CreateTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 (DEFAULT_TABLE table=test-table [id=12e057aacd124553b470fbff814f7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:56.689777 14762 tablet_service.cc:1467] Processing CreateTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 (DEFAULT_TABLE table=test-table [id=12e057aacd124553b470fbff814f7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:56.689954 14496 tablet_service.cc:1467] Processing CreateTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 (DEFAULT_TABLE table=test-table [id=12e057aacd124553b470fbff814f7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:56.690873 14762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f86bfa45dd4edc890cafbf6e072521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:56.690865 14629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f86bfa45dd4edc890cafbf6e072521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:56.690980 14496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f86bfa45dd4edc890cafbf6e072521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:56.696662 14845 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Bootstrap starting.
I20260927 00:33:56.697988 14846 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Bootstrap starting.
I20260927 00:33:56.697988 14847 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Bootstrap starting.
I20260927 00:33:56.699163 14845 tablet_bootstrap.cc:654] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:56.700138 14847 tablet_bootstrap.cc:654] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:56.700138 14846 tablet_bootstrap.cc:654] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:56.700300 14845 log.cc:824] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:56.701385 14847 log.cc:824] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:56.701385 14846 log.cc:824] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:56.702679 14845 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: No bootstrap required, opened a new log
I20260927 00:33:56.702863 14845 ts_tablet_manager.cc:1397] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:33:56.703715 14846 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: No bootstrap required, opened a new log
I20260927 00:33:56.703940 14846 ts_tablet_manager.cc:1397] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:33:56.704106 14847 tablet_bootstrap.cc:492] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: No bootstrap required, opened a new log
I20260927 00:33:56.704278 14847 ts_tablet_manager.cc:1397] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:33:56.706717 14845 raft_consensus.cc:348] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.706977 14845 raft_consensus.cc:374] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:56.707034 14845 raft_consensus.cc:729] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04da11291e9f487d97e974f89bcf1143, State: Initialized, Role: FOLLOWER
I20260927 00:33:56.706897 14846 raft_consensus.cc:348] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.707167 14846 raft_consensus.cc:374] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:56.707231 14846 raft_consensus.cc:729] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2238addbb8f74ab485146628d1f90a23, State: Initialized, Role: FOLLOWER
I20260927 00:33:56.707484 14845 consensus_queue.cc:260] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.707669 14846 consensus_queue.cc:260] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.708066 14847 raft_consensus.cc:348] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.708307 14847 raft_consensus.cc:374] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:56.708366 14847 raft_consensus.cc:729] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492d0fa602b1461aa4cd95579b741d23, State: Initialized, Role: FOLLOWER
I20260927 00:33:56.708603 14846 ts_tablet_manager.cc:1428] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:33:56.708709 14845 ts_tablet_manager.cc:1428] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:33:56.708746 14847 consensus_queue.cc:260] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.708951 14828 heartbeater.cc:503] Master 127.10.11.254:34653 was elected leader, sending a full tablet report...
I20260927 00:33:56.709016 14562 heartbeater.cc:503] Master 127.10.11.254:34653 was elected leader, sending a full tablet report...
I20260927 00:33:56.709347 14695 heartbeater.cc:503] Master 127.10.11.254:34653 was elected leader, sending a full tablet report...
I20260927 00:33:56.709650 14847 ts_tablet_manager.cc:1428] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 00:33:56.736986 14563 tablet.cc:2369] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:56.829058 14851 raft_consensus.cc:482] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:56.829288 14851 raft_consensus.cc:504] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.830415 14851 leader_election.cc:288] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2238addbb8f74ab485146628d1f90a23 (127.10.11.195:35447), 492d0fa602b1461aa4cd95579b741d23 (127.10.11.194:38461)
I20260927 00:33:56.834404 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.830839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52763 (local address 127.10.11.194:38461)
0927 00:33:56.830960 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.830965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.831113 (+ 148us) server_negotiation.cc:407] Connection header received
0927 00:33:56.831301 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.831306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.831363 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.831422 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:56.831744 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.832249 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833376 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.833776 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.833960 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.834214 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.834280 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:33:56.834405 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.830790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:59691 (local address 127.10.11.195:35447)
0927 00:33:56.830935 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.830939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.831198 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:33:56.831337 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.831339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.831380 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.831473 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:33:56.831780 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.832268 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833408 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.833814 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833950 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.834001 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.834199 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.834279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:33:56.834426 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.830664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:38461 (local address 127.10.11.193:52763)
0927 00:33:56.831022 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.831034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.831160 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.831466 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.831469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.831484 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:56.831631 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.831636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.832366 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.832369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.833244 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.833253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.833918 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.834029 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.834060 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.834195 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.834275 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":21,"thread_start_us":97,"threads_started":1}
I20260927 00:33:56.834431 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.830664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35447 (local address 127.10.11.193:59691)
0927 00:33:56.831101 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.831113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:56.831205 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:56.831502 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:56.831504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:56.831533 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:33:56.831643 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.831648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.832408 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.832410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.833244 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:56.833252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.833960 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.833964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:56.834061 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.834077 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 00:33:56.834182 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:33:56.834257 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":71,"spinlock_wait_cycles":768,"thread_start_us":150,"threads_started":1}
I20260927 00:33:56.835069 14649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f86bfa45dd4edc890cafbf6e072521" candidate_uuid: "04da11291e9f487d97e974f89bcf1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492d0fa602b1461aa4cd95579b741d23" is_pre_election: true
I20260927 00:33:56.835279 14782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f86bfa45dd4edc890cafbf6e072521" candidate_uuid: "04da11291e9f487d97e974f89bcf1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2238addbb8f74ab485146628d1f90a23" is_pre_election: true
I20260927 00:33:56.835382 14649 raft_consensus.cc:2454] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04da11291e9f487d97e974f89bcf1143 in term 0.
I20260927 00:33:56.835561 14782 raft_consensus.cc:2454] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04da11291e9f487d97e974f89bcf1143 in term 0.
I20260927 00:33:56.835901 14450 leader_election.cc:302] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da11291e9f487d97e974f89bcf1143, 492d0fa602b1461aa4cd95579b741d23; no voters:
I20260927 00:33:56.836201 14851 raft_consensus.cc:2781] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:56.836284 14851 raft_consensus.cc:482] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:56.836328 14851 raft_consensus.cc:3037] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:56.837319 14851 raft_consensus.cc:504] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.837637 14851 leader_election.cc:288] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [CANDIDATE]: Term 1 election: Requested vote from peers 2238addbb8f74ab485146628d1f90a23 (127.10.11.195:35447), 492d0fa602b1461aa4cd95579b741d23 (127.10.11.194:38461)
I20260927 00:33:56.838003 14782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f86bfa45dd4edc890cafbf6e072521" candidate_uuid: "04da11291e9f487d97e974f89bcf1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2238addbb8f74ab485146628d1f90a23"
I20260927 00:33:56.838161 14782 raft_consensus.cc:3037] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:56.838200 14649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f86bfa45dd4edc890cafbf6e072521" candidate_uuid: "04da11291e9f487d97e974f89bcf1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492d0fa602b1461aa4cd95579b741d23"
I20260927 00:33:56.838371 14649 raft_consensus.cc:3037] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:56.839257 14782 raft_consensus.cc:2454] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04da11291e9f487d97e974f89bcf1143 in term 1.
I20260927 00:33:56.839560 14649 raft_consensus.cc:2454] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04da11291e9f487d97e974f89bcf1143 in term 1.
I20260927 00:33:56.839572 14452 leader_election.cc:302] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da11291e9f487d97e974f89bcf1143, 2238addbb8f74ab485146628d1f90a23; no voters:
I20260927 00:33:56.839767 14851 raft_consensus.cc:2781] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:56.840026 14851 raft_consensus.cc:686] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [term 1 LEADER]: Becoming Leader. State: Replica: 04da11291e9f487d97e974f89bcf1143, State: Running, Role: LEADER
I20260927 00:33:56.840351 14851 consensus_queue.cc:237] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } }
I20260927 00:33:56.843523 14370 catalog_manager.cc:5495] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04da11291e9f487d97e974f89bcf1143 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "04da11291e9f487d97e974f89bcf1143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04da11291e9f487d97e974f89bcf1143" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 36221 } health_report { overall_health: HEALTHY } } }
I20260927 00:33:56.873426 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.869666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.10.11.193:36221)
0927 00:33:56.869810 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:56.869818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:56.869872 (+ 54us) server_negotiation.cc:407] Connection header received
0927 00:33:56.870077 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:56.870080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:56.870161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:56.870278 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:33:56.870554 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.871068 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.871952 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:56.872171 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:56.872283 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:56.872596 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:33:56.873030 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:33:56.873168 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:56.873229 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:33:56.873281 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:33:56.879133 14649 raft_consensus.cc:1261] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 1 FOLLOWER]: Refusing update from remote peer 04da11291e9f487d97e974f89bcf1143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:56.879133 14782 raft_consensus.cc:1261] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 1 FOLLOWER]: Refusing update from remote peer 04da11291e9f487d97e974f89bcf1143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:56.880050 14856 consensus_queue.cc:1035] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492d0fa602b1461aa4cd95579b741d23" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:56.880219 14851 consensus_queue.cc:1035] T a0f86bfa45dd4edc890cafbf6e072521 P 04da11291e9f487d97e974f89bcf1143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2238addbb8f74ab485146628d1f90a23" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:56.887356 14859 mvcc.cc:205] Tried to move back new op lower bound from 7333761994250022912 to 7333761994100613120. Current Snapshot: MvccSnapshot[applied={T|T < 7333761994250022912}]
I20260927 00:33:56.895745 14372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:56.895891 14372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:56.898167 14372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75ce0191566467da5cc91ab36d878ac: Sending DeleteTablet for 3 replicas of tablet a0f86bfa45dd4edc890cafbf6e072521
I20260927 00:33:56.898900 14762 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:56 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:42910
I20260927 00:33:56.898998 14496 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:56 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:51804
I20260927 00:33:56.899397 14869 tablet_replica.cc:330] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: stopping tablet replica
I20260927 00:33:56.899677 14869 raft_consensus.cc:2229] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:56.899701 14629 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f86bfa45dd4edc890cafbf6e072521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:56 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:52934
I20260927 00:33:56.900007 14869 raft_consensus.cc:2258] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:56.900038 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14434
I20260927 00:33:56.902626 14869 ts_tablet_manager.cc:1905] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:56.903091 14871 tablet_replica.cc:330] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: stopping tablet replica
I20260927 00:33:56.903359 14871 raft_consensus.cc:2229] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:56.903621 14871 raft_consensus.cc:2258] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:56.904946 14871 ts_tablet_manager.cc:1905] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:56.905185 14869 ts_tablet_manager.cc:1918] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:56.905289 14869 log.cc:1196] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/a0f86bfa45dd4edc890cafbf6e072521
I20260927 00:33:56.905642 14869 ts_tablet_manager.cc:1939] T a0f86bfa45dd4edc890cafbf6e072521 P 2238addbb8f74ab485146628d1f90a23: Deleting consensus metadata
I20260927 00:33:56.906863 14359 catalog_manager.cc:4841] TS 2238addbb8f74ab485146628d1f90a23 (127.10.11.195:35447): tablet a0f86bfa45dd4edc890cafbf6e072521 (table test-table [id=12e057aacd124553b470fbff814f7835]) successfully deleted
I20260927 00:33:56.907406 14871 ts_tablet_manager.cc:1918] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:56.907496 14871 log.cc:1196] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/a0f86bfa45dd4edc890cafbf6e072521
I20260927 00:33:56.907707 14871 ts_tablet_manager.cc:1939] T a0f86bfa45dd4edc890cafbf6e072521 P 492d0fa602b1461aa4cd95579b741d23: Deleting consensus metadata
I20260927 00:33:56.908650 14357 catalog_manager.cc:4841] TS 492d0fa602b1461aa4cd95579b741d23 (127.10.11.194:38461): tablet a0f86bfa45dd4edc890cafbf6e072521 (table test-table [id=12e057aacd124553b470fbff814f7835]) successfully deleted
I20260927 00:33:56.908784 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14567
W20260927 00:33:56.908679 14359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:36221 (error 108)
I20260927 00:33:56.914762 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14700
I20260927 00:33:56.915086 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:56.914737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:36221 (local address 127.0.0.1:51822)
0927 00:33:56.914905 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:56.914983 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:36221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 00:33:56.916283 14359 catalog_manager.cc:4584] TS 04da11291e9f487d97e974f89bcf1143 (127.10.11.193:36221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0f86bfa45dd4edc890cafbf6e072521: Network error: Client connection negotiation failed: client connection to 127.10.11.193:36221: connect: Connection refused (error 111)
I20260927 00:33:56.920184 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14341
2026-09-27T00:33:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2259 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:56Z Disabled control of system clock
I20260927 00:33:56.950049 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36165
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:33:57.056813 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:57.057036 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:57.057073 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:57.061025 14876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:57.061080 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:57.061100 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:57.061120 14876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:57.061178 14876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:33:57.065784 14876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:57.066744 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:57.067996 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:57.075793 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.075835 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.075827 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:57.076385 14876 server_base.cc:1034] running on GCE node
I20260927 00:33:57.076877 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:57.077889 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:57.079077 14876 hybrid_clock.cc:648] HybridClock initialized: now 1790469237079057 us; error 36 us; skew 500 ppm
I20260927 00:33:57.080911 14876 webserver.cc:460] Webserver started at http://127.10.11.254:37999/ using document root <none> and password file <none>
I20260927 00:33:57.081456 14876 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:57.081545 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:57.081773 14876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:57.083709 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "54b70638ffe64e77af2f4b0194201580"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "141d1cfbd9ab9ad727c725d7e2d08404"
server_key_iv: "e8cdd338d9669780f12b88a0cf91776c"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.084245 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "54b70638ffe64e77af2f4b0194201580"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "141d1cfbd9ab9ad727c725d7e2d08404"
server_key_iv: "e8cdd338d9669780f12b88a0cf91776c"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.087720 14876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:57.089953 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.091144 14876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:57.091305 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "54b70638ffe64e77af2f4b0194201580"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "141d1cfbd9ab9ad727c725d7e2d08404"
server_key_iv: "e8cdd338d9669780f12b88a0cf91776c"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.091413 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:57.096707 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:57.097247 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:57.097421 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:57.103883 14876 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:36165
I20260927 00:33:57.103919 14942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:36165 every 8 connection(s)
I20260927 00:33:57.104714 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:57.105082 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14876
I20260927 00:33:57.105229 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:57.105491 10287 external_mini_cluster.cc:1428] Setting key 3e3736d1f381b0fd0ded0ffdc8faae2e
I20260927 00:33:57.107686 14943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:57.111032 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.106766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.10.11.254:36165)
0927 00:33:57.107734 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.107750 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.107801 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:33:57.108404 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.108418 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.108723 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.108977 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.109763 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.109768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.109779 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.109788 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.109834 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.109848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.109966 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.110072 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.110362 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":525,"threads_started":1}
I20260927 00:33:57.113211 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: Bootstrap starting.
I20260927 00:33:57.115167 14943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:57.115990 14943 log.cc:824] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:57.117594 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: No bootstrap required, opened a new log
I20260927 00:33:57.121091 14943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } }
I20260927 00:33:57.121309 14943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:57.121384 14943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b70638ffe64e77af2f4b0194201580, State: Initialized, Role: FOLLOWER
I20260927 00:33:57.121743 14943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } }
I20260927 00:33:57.121883 14943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:57.121970 14943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:57.122066 14943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:57.122849 14943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } }
I20260927 00:33:57.123190 14943 leader_election.cc:302] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b70638ffe64e77af2f4b0194201580; no voters:
I20260927 00:33:57.123518 14943 leader_election.cc:288] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:57.123646 14948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:57.123888 14948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [term 1 LEADER]: Becoming Leader. State: Replica: 54b70638ffe64e77af2f4b0194201580, State: Running, Role: LEADER
I20260927 00:33:57.124186 14948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } }
I20260927 00:33:57.124630 14943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:57.125753 14949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b70638ffe64e77af2f4b0194201580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } } }
I20260927 00:33:57.125886 14949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:57.126195 14950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b70638ffe64e77af2f4b0194201580. Latest consensus state: current_term: 1 leader_uuid: "54b70638ffe64e77af2f4b0194201580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b70638ffe64e77af2f4b0194201580" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36165 } } }
I20260927 00:33:57.126272 14957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:57.126303 14950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:57.128934 14957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:57.133694 14957 catalog_manager.cc:1347] Generated new cluster ID: 982b76fb599c4b5bbc0e56cdfe9da6ab
I20260927 00:33:57.133760 14957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:57.145153 14957 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:57.145421 14957 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:57.157245 14957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: Generated new TSK 0
I20260927 00:33:57.157917 14957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:33:57.166875 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--builtin_ntp_servers=127.10.11.212:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:33:57.280059 14967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:57.280278 14967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:57.280314 14967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:57.284116 14967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:57.284159 14967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:57.284209 14967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:33:57.288686 14967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:57.289496 14967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:57.290745 14967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:57.299672 14975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.299700 14973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.299700 14972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:57.299870 14967 server_base.cc:1034] running on GCE node
I20260927 00:33:57.300415 14967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:57.301036 14967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:57.302196 14967 hybrid_clock.cc:648] HybridClock initialized: now 1790469237302170 us; error 48 us; skew 500 ppm
I20260927 00:33:57.304052 14967 webserver.cc:460] Webserver started at http://127.10.11.193:38941/ using document root <none> and password file <none>
I20260927 00:33:57.304616 14967 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:57.304709 14967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:57.304951 14967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:57.307039 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "a3e11c7abefe47049a9b683f2e4651ca"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "98d8e0de49963ea9a90eacbb0445eba9"
server_key_iv: "880d467ddc112c09b170e26e041cfba2"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.307618 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "a3e11c7abefe47049a9b683f2e4651ca"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "98d8e0de49963ea9a90eacbb0445eba9"
server_key_iv: "880d467ddc112c09b170e26e041cfba2"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.311363 14967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:33:57.313771 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.315008 14967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:33:57.315145 14967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "a3e11c7abefe47049a9b683f2e4651ca"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "98d8e0de49963ea9a90eacbb0445eba9"
server_key_iv: "880d467ddc112c09b170e26e041cfba2"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.315258 14967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:57.319861 14967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:57.320607 14967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:57.320847 14967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:57.321511 14967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:57.322531 14967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:57.322604 14967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.322671 14967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:57.322728 14967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.334287 14967 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:41063
I20260927 00:33:57.334307 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:41063 every 8 connection(s)
I20260927 00:33:57.335521 14967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:33:57.337462 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 14967
I20260927 00:33:57.337589 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:33:57.337932 10287 external_mini_cluster.cc:1428] Setting key b2f2caf463bc1483832486912e6fc183
I20260927 00:33:57.340056 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--builtin_ntp_servers=127.10.11.212:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:33:57.343034 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.337434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:47575 (local address 127.10.11.254:36165)
0927 00:33:57.337588 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.337595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.339038 (+ 1443us) server_negotiation.cc:407] Connection header received
0927 00:33:57.341003 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.341009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.341057 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.341153 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.342441 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.342447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.342449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.342464 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.342489 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.342501 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.342606 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.342830 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.342877 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:33:57.344077 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.337940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36165 (local address 127.10.11.193:47575)
0927 00:33:57.338847 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.338902 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.340727 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.341346 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.341358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.341485 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.342084 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.342087 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.342165 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.342268 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.342611 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.342623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.342632 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.342819 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.343055 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":182,"threads_started":1}
I20260927 00:33:57.345237 15095 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36165
I20260927 00:33:57.345467 15095 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:57.345839 15095 heartbeater.cc:511] Master 127.10.11.254:36165 requested a full tablet report, sending...
I20260927 00:33:57.347348 14907 ts_manager.cc:194] Registered new tserver with Master: a3e11c7abefe47049a9b683f2e4651ca (127.10.11.193:41063)
W20260927 00:33:57.447041 15098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:57.447249 15098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:57.447285 15098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:57.451165 15098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:57.451206 15098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:57.451284 15098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:33:57.456679 15098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:57.457545 15098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:57.458755 15098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:57.467447 15103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.467523 15106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.467475 15104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:57.467602 15098 server_base.cc:1034] running on GCE node
I20260927 00:33:57.468053 15098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:57.468590 15098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:57.469796 15098 hybrid_clock.cc:648] HybridClock initialized: now 1790469237469756 us; error 32 us; skew 500 ppm
I20260927 00:33:57.471537 15098 webserver.cc:460] Webserver started at http://127.10.11.194:46755/ using document root <none> and password file <none>
I20260927 00:33:57.472060 15098 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:57.472148 15098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:57.472358 15098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:57.474237 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "14bf40e5be354c3795d9849f82be3359"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "f2a77e74d2f4c950e38cb58c4433e1c1"
server_key_iv: "6a402d31f44c42b6539fce33d4325aa8"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.474763 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "14bf40e5be354c3795d9849f82be3359"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "f2a77e74d2f4c950e38cb58c4433e1c1"
server_key_iv: "6a402d31f44c42b6539fce33d4325aa8"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.478050 15098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:33:57.480289 15112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.481520 15098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:33:57.481662 15098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "14bf40e5be354c3795d9849f82be3359"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "f2a77e74d2f4c950e38cb58c4433e1c1"
server_key_iv: "6a402d31f44c42b6539fce33d4325aa8"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.481791 15098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:57.486027 15098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:57.486578 15098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:57.486756 15098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:57.487231 15098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:57.488052 15098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:57.488122 15098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.488184 15098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:57.488240 15098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.496492 15098 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:42473
I20260927 00:33:57.496575 15225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:42473 every 8 connection(s)
I20260927 00:33:57.497310 15098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:33:57.502782 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.499000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40441 (local address 127.10.11.254:36165)
0927 00:33:57.499128 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.499138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.500044 (+ 906us) server_negotiation.cc:407] Connection header received
0927 00:33:57.501229 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.501232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.501268 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.501347 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.502423 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.502426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.502428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.502439 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.502457 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.502465 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.502539 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.502651 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.502685 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:33:57.503562 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.499330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36165 (local address 127.10.11.194:40441)
0927 00:33:57.499901 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.499936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.501043 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.501496 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.501501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.501583 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.502160 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.502162 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.502214 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.502285 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.502539 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.502545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.502549 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.502667 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.502872 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260927 00:33:57.504637 15226 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36165
I20260927 00:33:57.504892 15226 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:57.505259 15226 heartbeater.cc:511] Master 127.10.11.254:36165 requested a full tablet report, sending...
I20260927 00:33:57.506240 14907 ts_manager.cc:194] Registered new tserver with Master: 14bf40e5be354c3795d9849f82be3359 (127.10.11.194:42473)
I20260927 00:33:57.506364 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15098
I20260927 00:33:57.506476 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:33:57.506788 10287 external_mini_cluster.cc:1428] Setting key d88d545ef8dee37ac9a69fa66e19cbeb
I20260927 00:33:57.508535 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--builtin_ntp_servers=127.10.11.212:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:33:57.614296 15229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:57.614511 15229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:57.614576 15229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:57.618511 15229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:57.618557 15229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:57.618609 15229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:33:57.623287 15229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:57.624203 15229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:57.625602 15229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:57.634707 15237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.634843 15234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:57.634737 15235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:57.635067 15229 server_base.cc:1034] running on GCE node
I20260927 00:33:57.635470 15229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:57.636040 15229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:57.637221 15229 hybrid_clock.cc:648] HybridClock initialized: now 1790469237637189 us; error 45 us; skew 500 ppm
I20260927 00:33:57.639006 15229 webserver.cc:460] Webserver started at http://127.10.11.195:44779/ using document root <none> and password file <none>
I20260927 00:33:57.639547 15229 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:57.639635 15229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:57.639853 15229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:57.641734 15229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "dcd68da63e7c45038276b02ee8b7d120"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "547f0f13f57edf59c8289e925d6612f8"
server_key_iv: "f75f0e8607168d297688b4c58c4b6c94"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.642357 15229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "dcd68da63e7c45038276b02ee8b7d120"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "547f0f13f57edf59c8289e925d6612f8"
server_key_iv: "f75f0e8607168d297688b4c58c4b6c94"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.646037 15229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:33:57.648424 15243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.649631 15229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:33:57.649778 15229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "dcd68da63e7c45038276b02ee8b7d120"
format_stamp: "Formatted at 2026-09-27 00:33:57 on dist-test-slave-6gxc"
server_key: "547f0f13f57edf59c8289e925d6612f8"
server_key_iv: "f75f0e8607168d297688b4c58c4b6c94"
server_key_version: "encryptionkey@0"
I20260927 00:33:57.649915 15229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:57.654141 15229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:57.654688 15229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:57.654871 15229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:57.655355 15229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:33:57.656169 15229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:33:57.656239 15229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.656298 15229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:33:57.656348 15229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:57.664723 15229 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:37163
I20260927 00:33:57.664791 15356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:37163 every 8 connection(s)
I20260927 00:33:57.665531 15229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:33:57.670842 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.667245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:32815 (local address 127.10.11.254:36165)
0927 00:33:57.667381 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.667384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.668197 (+ 813us) server_negotiation.cc:407] Connection header received
0927 00:33:57.669400 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.669403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.669436 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.669505 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.670469 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.670472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.670474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.670485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.670502 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.670510 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.670579 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.670692 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.670725 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:33:57.671576 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.667505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36165 (local address 127.10.11.195:32815)
0927 00:33:57.668081 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.668118 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.669226 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.669619 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.669624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.669702 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.670241 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.670242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.670281 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.670341 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.670584 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.670589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.670593 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.670710 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.670902 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260927 00:33:57.672590 15357 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36165
I20260927 00:33:57.672835 15357 heartbeater.cc:464] Registering TS with master...
I20260927 00:33:57.673197 15357 heartbeater.cc:511] Master 127.10.11.254:36165 requested a full tablet report, sending...
I20260927 00:33:57.674160 14907 ts_manager.cc:194] Registered new tserver with Master: dcd68da63e7c45038276b02ee8b7d120 (127.10.11.195:37163)
I20260927 00:33:57.674312 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15229
I20260927 00:33:57.674422 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:33:57.674700 10287 external_mini_cluster.cc:1428] Setting key 7e552539df54f573e202b4b8774c38d2
I20260927 00:33:57.676558 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:33:57.679644 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.678384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.10.11.254:36165)
0927 00:33:57.678494 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.678497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.678664 (+ 167us) server_negotiation.cc:407] Connection header received
0927 00:33:57.678741 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.678743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.678796 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.678859 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.679394 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.679397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.679399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.679408 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.679422 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.679429 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.679497 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.679540 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.679567 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:33:57.682408 14907 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:33:57.685158 14907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:33:57.696087 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.693498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.10.11.194:42473)
0927 00:33:57.693943 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.693955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.693980 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:33:57.694036 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.694043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.694269 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.694468 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.695532 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.695536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.695545 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.695554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.695587 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.695605 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.695683 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.695903 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.695971 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":131,"threads_started":1}
I20260927 00:33:57.696102 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.692796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:41063 (local address 127.0.0.1:33742)
0927 00:33:57.693342 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.693397 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.693487 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.694726 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.694731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.694752 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.695243 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.695245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.695265 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.695357 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.695902 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.695905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.695909 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.695963 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.695990 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":65,"threads_started":1}
I20260927 00:33:57.696102 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.692948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:42473 (local address 127.0.0.1:41248)
0927 00:33:57.693618 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.693631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.693695 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.694480 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.694485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.694558 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.695244 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.695247 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.695265 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.695374 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.695692 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.695700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.695713 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.695870 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.695938 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":116,"threads_started":1}
I20260927 00:33:57.696827 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.692943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.10.11.193:41063)
0927 00:33:57.694294 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.694307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.694333 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:33:57.694392 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.694400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.694568 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.694969 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.695694 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.695700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.695711 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.695722 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.695764 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.695793 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.696510 (+ 717us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.696605 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.696670 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":85,"threads_started":1}
I20260927 00:33:57.696995 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.693241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:37163 (local address 127.0.0.1:36214)
0927 00:33:57.694006 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.694026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.694143 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.695217 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.695220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.695234 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.696288 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.696290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.696300 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.696331 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.696736 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.696739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.696742 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.696810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.696857 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":109,"threads_started":1}
I20260927 00:33:57.697335 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.693674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.10.11.195:37163)
0927 00:33:57.693952 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.693965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.694234 (+ 269us) server_negotiation.cc:407] Connection header received
0927 00:33:57.694283 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.694288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.694479 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.694597 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.696453 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.696458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.696468 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.696522 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.696563 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.696595 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.696696 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.697140 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.697207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260927 00:33:57.699110 15160 tablet_service.cc:1467] Processing CreateTablet for tablet 9b85e13ea761432aaca206b886206f4b (DEFAULT_TABLE table=test-table [id=301839f85d1f4f15a770bc25e5dbf719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:57.699424 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 9b85e13ea761432aaca206b886206f4b (DEFAULT_TABLE table=test-table [id=301839f85d1f4f15a770bc25e5dbf719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:57.699574 15024 tablet_service.cc:1467] Processing CreateTablet for tablet 9b85e13ea761432aaca206b886206f4b (DEFAULT_TABLE table=test-table [id=301839f85d1f4f15a770bc25e5dbf719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:33:57.700203 15160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b85e13ea761432aaca206b886206f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:57.700361 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b85e13ea761432aaca206b886206f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:57.700477 15024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b85e13ea761432aaca206b886206f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:57.705948 15374 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Bootstrap starting.
I20260927 00:33:57.707667 15376 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Bootstrap starting.
I20260927 00:33:57.708039 15374 tablet_bootstrap.cc:654] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:57.708702 15374 log.cc:824] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:57.709386 15375 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Bootstrap starting.
I20260927 00:33:57.709542 15376 tablet_bootstrap.cc:654] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:57.710351 15376 log.cc:824] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:57.711251 15375 tablet_bootstrap.cc:654] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:57.711413 15374 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: No bootstrap required, opened a new log
I20260927 00:33:57.711601 15374 ts_tablet_manager.cc:1397] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 00:33:57.711947 15375 log.cc:824] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:57.712220 15376 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: No bootstrap required, opened a new log
I20260927 00:33:57.712404 15376 ts_tablet_manager.cc:1397] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:33:57.713459 15375 tablet_bootstrap.cc:492] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: No bootstrap required, opened a new log
I20260927 00:33:57.713647 15375 ts_tablet_manager.cc:1397] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 00:33:57.716581 15374 raft_consensus.cc:348] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.716868 15374 raft_consensus.cc:374] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:57.716764 15376 raft_consensus.cc:348] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.716935 15374 raft_consensus.cc:729] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd68da63e7c45038276b02ee8b7d120, State: Initialized, Role: FOLLOWER
I20260927 00:33:57.716974 15376 raft_consensus.cc:374] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:57.717032 15376 raft_consensus.cc:729] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14bf40e5be354c3795d9849f82be3359, State: Initialized, Role: FOLLOWER
I20260927 00:33:57.717008 15375 raft_consensus.cc:348] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.717274 15375 raft_consensus.cc:374] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:57.717340 15375 raft_consensus.cc:729] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e11c7abefe47049a9b683f2e4651ca, State: Initialized, Role: FOLLOWER
I20260927 00:33:57.717402 15376 consensus_queue.cc:260] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.717392 15374 consensus_queue.cc:260] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.717752 15375 consensus_queue.cc:260] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.718302 15357 heartbeater.cc:503] Master 127.10.11.254:36165 was elected leader, sending a full tablet report...
I20260927 00:33:57.718422 15376 ts_tablet_manager.cc:1428] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:33:57.718492 15374 ts_tablet_manager.cc:1428] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:33:57.718662 15375 ts_tablet_manager.cc:1428] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:33:57.718801 15226 heartbeater.cc:503] Master 127.10.11.254:36165 was elected leader, sending a full tablet report...
I20260927 00:33:57.719061 15095 heartbeater.cc:503] Master 127.10.11.254:36165 was elected leader, sending a full tablet report...
W20260927 00:33:57.748189 15227 tablet.cc:2369] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:33:57.836772 15096 tablet.cc:2369] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:57.904915 15382 raft_consensus.cc:482] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:33:57.905166 15382 raft_consensus.cc:504] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.906301 15382 leader_election.cc:288] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14bf40e5be354c3795d9849f82be3359 (127.10.11.194:42473), dcd68da63e7c45038276b02ee8b7d120 (127.10.11.195:37163)
I20260927 00:33:57.908783 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.906692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:39849 (local address 127.10.11.195:37163)
0927 00:33:57.906961 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.906965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.907206 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:33:57.907252 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.907255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.907311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.907388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.908324 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.908328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.908331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.908345 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.908369 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.908381 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.908483 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.908592 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.908635 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260927 00:33:57.908805 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.906714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:45437 (local address 127.10.11.194:42473)
0927 00:33:57.906853 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.906858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.907013 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:33:57.907220 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.907224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.907269 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.907361 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.908324 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.908328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.908331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.908345 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.908369 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.908381 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.908483 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.908592 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.908635 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:33:57.908807 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.906592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:37163 (local address 127.10.11.193:39849)
0927 00:33:57.907023 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.907044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.907119 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.907440 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.907444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.907485 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.908151 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.908152 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.908165 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.908181 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.908482 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.908485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.908488 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.908575 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.908627 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":44,"thread_start_us":106,"threads_started":1}
I20260927 00:33:57.908807 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.906592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:42473 (local address 127.10.11.193:45437)
0927 00:33:57.906919 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.906937 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:57.907079 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:57.907440 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:57.907444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:57.907476 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:33:57.908115 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:33:57.908118 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:33:57.908133 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:33:57.908165 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:33:57.908482 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:33:57.908485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:33:57.908488 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:33:57.908575 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:33:57.908637 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":18,"thread_start_us":137,"threads_started":1}
I20260927 00:33:57.909526 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b85e13ea761432aaca206b886206f4b" candidate_uuid: "a3e11c7abefe47049a9b683f2e4651ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bf40e5be354c3795d9849f82be3359" is_pre_election: true
I20260927 00:33:57.909524 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b85e13ea761432aaca206b886206f4b" candidate_uuid: "a3e11c7abefe47049a9b683f2e4651ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd68da63e7c45038276b02ee8b7d120" is_pre_election: true
I20260927 00:33:57.909825 15180 raft_consensus.cc:2454] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e11c7abefe47049a9b683f2e4651ca in term 0.
I20260927 00:33:57.909843 15311 raft_consensus.cc:2454] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e11c7abefe47049a9b683f2e4651ca in term 0.
I20260927 00:33:57.910274 14982 leader_election.cc:302] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14bf40e5be354c3795d9849f82be3359, a3e11c7abefe47049a9b683f2e4651ca; no voters:
I20260927 00:33:57.910542 15382 raft_consensus.cc:2781] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:33:57.910624 15382 raft_consensus.cc:482] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:33:57.910683 15382 raft_consensus.cc:3037] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:57.911778 15382 raft_consensus.cc:504] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
I20260927 00:33:57.912201 15382 leader_election.cc:288] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [CANDIDATE]: Term 1 election: Requested vote from peers 14bf40e5be354c3795d9849f82be3359 (127.10.11.194:42473), dcd68da63e7c45038276b02ee8b7d120 (127.10.11.195:37163)
I20260927 00:33:57.912518 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b85e13ea761432aaca206b886206f4b" candidate_uuid: "a3e11c7abefe47049a9b683f2e4651ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bf40e5be354c3795d9849f82be3359"
I20260927 00:33:57.912526 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b85e13ea761432aaca206b886206f4b" candidate_uuid: "a3e11c7abefe47049a9b683f2e4651ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd68da63e7c45038276b02ee8b7d120"
I20260927 00:33:57.912649 15311 raft_consensus.cc:3037] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:57.912650 15180 raft_consensus.cc:3037] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:57.913949 15180 raft_consensus.cc:2454] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e11c7abefe47049a9b683f2e4651ca in term 1.
I20260927 00:33:57.913949 15311 raft_consensus.cc:2454] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e11c7abefe47049a9b683f2e4651ca in term 1.
I20260927 00:33:57.914273 14985 leader_election.cc:302] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e11c7abefe47049a9b683f2e4651ca, dcd68da63e7c45038276b02ee8b7d120; no voters:
I20260927 00:33:57.914510 15382 raft_consensus.cc:2781] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:57.914696 15382 raft_consensus.cc:686] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [term 1 LEADER]: Becoming Leader. State: Replica: a3e11c7abefe47049a9b683f2e4651ca, State: Running, Role: LEADER
I20260927 00:33:57.915014 15382 consensus_queue.cc:237] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } }
W20260927 00:33:57.916397 15358 tablet.cc:2369] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:33:57.918288 14907 catalog_manager.cc:5495] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e11c7abefe47049a9b683f2e4651ca (127.10.11.193). New cstate: current_term: 1 leader_uuid: "a3e11c7abefe47049a9b683f2e4651ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e11c7abefe47049a9b683f2e4651ca" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 41063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 } health_report { overall_health: UNKNOWN } } }
I20260927 00:33:57.929579 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.928128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.10.11.193:41063)
0927 00:33:57.928250 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:57.928254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:57.928281 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:33:57.928415 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:57.928418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:57.928468 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:57.928533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:57.929143 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:57.929146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:57.929148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:57.929158 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:57.929175 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:57.929203 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:57.929308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:57.929372 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 00:33:57.929404 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:33:57.935181 15311 raft_consensus.cc:1261] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 1 FOLLOWER]: Refusing update from remote peer a3e11c7abefe47049a9b683f2e4651ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:57.935181 15180 raft_consensus.cc:1261] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 1 FOLLOWER]: Refusing update from remote peer a3e11c7abefe47049a9b683f2e4651ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:33:57.935943 15385 consensus_queue.cc:1035] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14bf40e5be354c3795d9849f82be3359" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:57.936160 15382 consensus_queue.cc:1035] T 9b85e13ea761432aaca206b886206f4b P a3e11c7abefe47049a9b683f2e4651ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd68da63e7c45038276b02ee8b7d120" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:33:57.944597 15389 mvcc.cc:205] Tried to move back new op lower bound from 7333761998575165440 to 7333761998502322176. Current Snapshot: MvccSnapshot[applied={T|T < 7333761998575165440}]
I20260927 00:33:57.952003 14906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:57.952129 14906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:33:57.954437 14906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54b70638ffe64e77af2f4b0194201580: Sending DeleteTablet for 3 replicas of tablet 9b85e13ea761432aaca206b886206f4b
I20260927 00:33:57.955163 15160 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b85e13ea761432aaca206b886206f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:57 UTC) from {username='slave'} at 127.0.0.1:41248
I20260927 00:33:57.955371 15291 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b85e13ea761432aaca206b886206f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:57 UTC) from {username='slave'} at 127.0.0.1:36214
I20260927 00:33:57.955621 15024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b85e13ea761432aaca206b886206f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:33:57 UTC) from {username='slave'} at 127.0.0.1:33742
I20260927 00:33:57.955953 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14967
I20260927 00:33:57.956045 15398 tablet_replica.cc:330] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: stopping tablet replica
I20260927 00:33:57.956674 15398 raft_consensus.cc:2229] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:57.956947 15398 raft_consensus.cc:2258] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:57.957895 15399 tablet_replica.cc:330] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: stopping tablet replica
I20260927 00:33:57.958171 15399 raft_consensus.cc:2229] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:33:57.958416 15399 raft_consensus.cc:2258] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:33:57.958494 15398 ts_tablet_manager.cc:1905] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:57.959455 15399 ts_tablet_manager.cc:1905] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:33:57.961432 15399 ts_tablet_manager.cc:1918] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:57.961539 15399 log.cc:1196] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/9b85e13ea761432aaca206b886206f4b
I20260927 00:33:57.961817 15399 ts_tablet_manager.cc:1939] T 9b85e13ea761432aaca206b886206f4b P dcd68da63e7c45038276b02ee8b7d120: Deleting consensus metadata
I20260927 00:33:57.962221 15398 ts_tablet_manager.cc:1918] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:33:57.962329 15398 log.cc:1196] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/9b85e13ea761432aaca206b886206f4b
I20260927 00:33:57.962625 15398 ts_tablet_manager.cc:1939] T 9b85e13ea761432aaca206b886206f4b P 14bf40e5be354c3795d9849f82be3359: Deleting consensus metadata
I20260927 00:33:57.962982 14894 catalog_manager.cc:4841] TS dcd68da63e7c45038276b02ee8b7d120 (127.10.11.195:37163): tablet 9b85e13ea761432aaca206b886206f4b (table test-table [id=301839f85d1f4f15a770bc25e5dbf719]) successfully deleted
I20260927 00:33:57.963721 14891 catalog_manager.cc:4841] TS 14bf40e5be354c3795d9849f82be3359 (127.10.11.194:42473): tablet 9b85e13ea761432aaca206b886206f4b (table test-table [id=301839f85d1f4f15a770bc25e5dbf719]) successfully deleted
W20260927 00:33:57.963928 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:41063 (error 108)
I20260927 00:33:57.964558 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15098
I20260927 00:33:57.964941 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:57.964547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:41063 (local address 127.0.0.1:33770)
0927 00:33:57.964717 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:57.964841 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:41063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:33:57.965461 14893 catalog_manager.cc:4584] TS a3e11c7abefe47049a9b683f2e4651ca (127.10.11.193:41063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b85e13ea761432aaca206b886206f4b: Network error: Client connection negotiation failed: client connection to 127.10.11.193:41063: connect: Connection refused (error 111)
I20260927 00:33:57.970224 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15229
I20260927 00:33:57.975695 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 14876
2026-09-27T00:33:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1054 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:33:57.996798 10287 mini_oidc.cc:152] Starting JWKS server
I20260927 00:33:57.998342 10287 webserver.cc:460] Webserver started at https://127.0.0.1:42709/ using document root <none> and password file <none>
I20260927 00:33:57.998512 10287 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:33:57.998795 10287 webserver.cc:460] Webserver started at http://127.0.0.1:44155/ using document root <none> and password file <none>
2026-09-27T00:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:58Z Disabled control of system clock
I20260927 00:33:58.007117 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43151
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:43151
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42709/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/testchainca.cert with env {}
W20260927 00:33:58.112936 15407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:58.113179 15407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:58.113255 15407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:58.117215 15407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:58.117303 15407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:58.117331 15407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:58.117367 15407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:58.117405 15407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:33:58.117440 15407 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:33:58.117475 15407 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42709/jwks.json
I20260927 00:33:58.122700 15407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:43151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42709/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:58.123607 15407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:58.124532 15407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:58.132622 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:58.132624 15412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:58.132740 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:58.133006 15407 server_base.cc:1034] running on GCE node
I20260927 00:33:58.133536 15407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:58.134609 15407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:58.135798 15407 hybrid_clock.cc:648] HybridClock initialized: now 1790469238135790 us; error 31 us; skew 500 ppm
I20260927 00:33:58.137492 15407 webserver.cc:460] Webserver started at http://127.10.11.254:45057/ using document root <none> and password file <none>
I20260927 00:33:58.138023 15407 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:58.138139 15407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:58.138345 15407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:58.140164 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "387e25277b984fd78f615f94c36c34e6"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "dc2c587ace942e6c8548f8c71f675b36"
server_key_iv: "9edf167617e2343c4c9a8bf7bdbbdc3a"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.140658 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "387e25277b984fd78f615f94c36c34e6"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "dc2c587ace942e6c8548f8c71f675b36"
server_key_iv: "9edf167617e2343c4c9a8bf7bdbbdc3a"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.143944 15407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:58.146050 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:58.147197 15407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:58.147296 15407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "387e25277b984fd78f615f94c36c34e6"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "dc2c587ace942e6c8548f8c71f675b36"
server_key_iv: "9edf167617e2343c4c9a8bf7bdbbdc3a"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.147369 15407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:58.166320 15407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:58.166839 15407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:58.166970 15407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:58.173507 15407 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:43151
I20260927 00:33:58.173524 15473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:43151 every 8 connection(s)
I20260927 00:33:58.174363 15407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:58.177148 15474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:58.181903 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15407
I20260927 00:33:58.182029 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:58.182329 10287 external_mini_cluster.cc:1428] Setting key f6067250e4be0446af62d2ed354d711c
I20260927 00:33:58.182551 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: Bootstrap starting.
I20260927 00:33:58.184924 15474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:58.185617 15474 log.cc:824] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:58.187356 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: No bootstrap required, opened a new log
I20260927 00:33:58.190585 15474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } }
I20260927 00:33:58.190862 15474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:58.190943 15474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387e25277b984fd78f615f94c36c34e6, State: Initialized, Role: FOLLOWER
I20260927 00:33:58.191412 15474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } }
I20260927 00:33:58.191535 15474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:58.191617 15474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:58.191684 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.183611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.10.11.254:43151)
0927 00:33:58.184262 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.184278 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.184313 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:33:58.184942 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.184958 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.185388 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.185721 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:58.186211 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.187144 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.188198 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.188693 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.188876 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.189804 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:58.189809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:58.189821 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:58.190262 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:58.190316 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:58.190340 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:58.190450 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:58.190588 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:33:58.190853 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260927 00:33:58.191741 15474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:58.192546 15474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } }
I20260927 00:33:58.192870 15474 leader_election.cc:302] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 387e25277b984fd78f615f94c36c34e6; no voters:
I20260927 00:33:58.193162 15474 leader_election.cc:288] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:58.193225 15479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:58.193413 15479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [term 1 LEADER]: Becoming Leader. State: Replica: 387e25277b984fd78f615f94c36c34e6, State: Running, Role: LEADER
I20260927 00:33:58.193883 15479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } }
I20260927 00:33:58.194365 15474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:58.194658 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:33:58.195436 15481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 387e25277b984fd78f615f94c36c34e6. Latest consensus state: current_term: 1 leader_uuid: "387e25277b984fd78f615f94c36c34e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } } }
I20260927 00:33:58.195992 15481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:58.195858 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "387e25277b984fd78f615f94c36c34e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387e25277b984fd78f615f94c36c34e6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43151 } } }
I20260927 00:33:58.196764 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6 [sys.catalog]: This master's current role is: LEADER
W20260927 00:33:58.199023 15494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:33:58.199101 15494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:33:58.199174 15493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:58.199834 15493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:58.204528 15493 catalog_manager.cc:1347] Generated new cluster ID: ccd7a6062fc643da912b2c617230a2d8
I20260927 00:33:58.204586 15493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:58.212102 15493 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:58.213208 15493 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:58.233076 15493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 387e25277b984fd78f615f94c36c34e6: Generated new TSK 0
I20260927 00:33:58.233946 15493 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:33:58.263790 15504 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:33:58.263906 15504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:33:58.264389 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.247496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.10.11.254:43151)
0927 00:33:58.247620 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.247625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.247880 (+ 255us) server_negotiation.cc:407] Connection header received
0927 00:33:58.248060 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.248063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.248118 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.248227 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:33:58.248442 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.249283 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.250163 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.250523 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.250625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.250995 (+ 370us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:33:58.264036 (+ 13041us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:33:58.264254 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:33:58.264467 15477 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:33:58.265487 15503 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:33:58.247411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43151 (local address 127.0.0.1:53096)
0927 00:33:58.247796 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:58.247809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:58.247914 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:58.248240 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:58.248242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:58.248248 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:33:58.248346 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.248350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.249398 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.249400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.250014 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.250021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.250663 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.250667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.250747 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.250815 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:33:58.264225 (+ 13410us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:33:58.264403 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.11.254:43151: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":161,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.11.254:43151: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:33:58.266422 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15407
2026-09-27T00:33:58Z chronyd exiting
I20260927 00:33:58.303454 10287 test_util.cc:175] -----------------------------------------------
I20260927 00:33:58.303540 10287 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469213433099-10287-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:33:58.304523 10287 mini_oidc.cc:152] Starting JWKS server
I20260927 00:33:58.305326 10287 webserver.cc:460] Webserver started at https://127.0.0.1:36435/ using document root <none> and password file <none>
I20260927 00:33:58.305450 10287 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:33:58.305734 10287 webserver.cc:460] Webserver started at http://127.0.0.1:46849/ using document root <none> and password file <none>
2026-09-27T00:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:58Z Disabled control of system clock
I20260927 00:33:58.314179 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42273
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36435/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/testchainca.cert with env {}
W20260927 00:33:58.417050 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:58.417270 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:58.417308 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:58.421208 15510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:58.421257 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:58.421273 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:58.421289 15510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:58.421303 15510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:33:58.421319 15510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:33:58.421332 15510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36435/jwks.json
I20260927 00:33:58.426427 15510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36435/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:58.427255 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:58.428094 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:58.435909 15515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:58.436019 15516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:58.435907 15518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:58.436133 15510 server_base.cc:1034] running on GCE node
I20260927 00:33:58.436641 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:58.437844 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:58.439066 15510 hybrid_clock.cc:648] HybridClock initialized: now 1790469238439050 us; error 30 us; skew 500 ppm
I20260927 00:33:58.440825 15510 webserver.cc:460] Webserver started at http://127.10.11.254:34099/ using document root <none> and password file <none>
I20260927 00:33:58.441375 15510 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:58.441464 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:58.441679 15510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:58.443606 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "cea36956428542a094ec662cdc10f189"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "15e458f61810a22f97053042c463091f"
server_key_iv: "8f9c61aa94493333200d216260813ef9"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.444152 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "cea36956428542a094ec662cdc10f189"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "15e458f61810a22f97053042c463091f"
server_key_iv: "8f9c61aa94493333200d216260813ef9"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.447600 15510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:58.449885 15524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:58.451049 15510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:58.451184 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "cea36956428542a094ec662cdc10f189"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "15e458f61810a22f97053042c463091f"
server_key_iv: "8f9c61aa94493333200d216260813ef9"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.451293 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:58.476864 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:58.477473 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:58.477660 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:58.484457 15510 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42273
I20260927 00:33:58.484468 15576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42273 every 8 connection(s)
I20260927 00:33:58.485317 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:58.488284 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:58.489014 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15510
I20260927 00:33:58.489146 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:58.489466 10287 external_mini_cluster.cc:1428] Setting key 3fce72dc323a8805bd2f1a68ee492335
I20260927 00:33:58.494153 15577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189: Bootstrap starting.
I20260927 00:33:58.496558 15577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:58.497340 15577 log.cc:824] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:58.498183 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.490690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.10.11.254:42273)
0927 00:33:58.491254 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.491270 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.491303 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:33:58.491941 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.491957 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.492320 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.492579 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:58.492900 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.493799 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.494995 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.495430 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.495549 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.496395 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:58.496400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:58.496411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:58.496841 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:58.496897 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:58.496916 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:58.496996 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:58.497155 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 00:33:58.497443 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":120,"threads_started":1}
I20260927 00:33:58.499248 15577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189: No bootstrap required, opened a new log
I20260927 00:33:58.500882 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:33:58.503330 15577 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } }
I20260927 00:33:58.503525 15577 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:58.503557 15577 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea36956428542a094ec662cdc10f189, State: Initialized, Role: FOLLOWER
I20260927 00:33:58.503877 15577 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } }
I20260927 00:33:58.503981 15577 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:58.504017 15577 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:58.504081 15577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:58.504812 15577 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } }
I20260927 00:33:58.505088 15577 leader_election.cc:302] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea36956428542a094ec662cdc10f189; no voters:
I20260927 00:33:58.505316 15577 leader_election.cc:288] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:58.505445 15582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:58.505671 15582 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [term 1 LEADER]: Becoming Leader. State: Replica: cea36956428542a094ec662cdc10f189, State: Running, Role: LEADER
I20260927 00:33:58.505950 15582 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } }
I20260927 00:33:58.506493 15577 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:58.507568 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea36956428542a094ec662cdc10f189. Latest consensus state: current_term: 1 leader_uuid: "cea36956428542a094ec662cdc10f189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } } }
I20260927 00:33:58.507699 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:58.507948 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea36956428542a094ec662cdc10f189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea36956428542a094ec662cdc10f189" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42273 } } }
I20260927 00:33:58.508037 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:58.508149 15591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:58.511049 15591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:58.516656 15591 catalog_manager.cc:1347] Generated new cluster ID: 9126963f4a804caa8ae140d771a7e39b
I20260927 00:33:58.516749 15591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:58.523195 15591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:58.524248 15591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:58.538837 15591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea36956428542a094ec662cdc10f189: Generated new TSK 0
I20260927 00:33:58.539516 15591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:33:58.565336 15607 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:33:58.565409 15607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:33:58.565948 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.550559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.10.11.254:42273)
0927 00:33:58.550689 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.550693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.550901 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:33:58.551040 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.551042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.551086 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.551172 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:33:58.551417 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.551936 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.553123 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.553348 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.553425 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.553742 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:33:58.565659 (+ 11917us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:33:58.565848 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:33:58.566022 15580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:33:58.566145 15606 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:33:58.550494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42273 (local address 127.0.0.1:39046)
0927 00:33:58.550810 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:58.550824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:58.550930 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:58.551189 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:58.551192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:58.551200 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:33:58.551318 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.551324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.552033 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.552036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.553005 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.553015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.553456 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.553460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.553552 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.553577 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:33:58.565810 (+ 12233us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:33:58.565962 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.11.254:42273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
I20260927 00:33:58.566692 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15510
2026-09-27T00:33:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:33:58.604662 10287 mini_oidc.cc:152] Starting JWKS server
I20260927 00:33:58.605533 10287 webserver.cc:460] Webserver started at https://127.0.0.1:37381/ using document root <none> and password file <none>
I20260927 00:33:58.605640 10287 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:33:58.605937 10287 webserver.cc:460] Webserver started at http://127.0.0.1:44169/ using document root <none> and password file <none>
2026-09-27T00:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:58Z Disabled control of system clock
I20260927 00:33:58.614342 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:40273
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:40273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:33:58.718191 15613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:58.718410 15613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:58.718446 15613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:58.722265 15613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:58.722309 15613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:58.722324 15613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:58.722340 15613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:58.722357 15613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:33:58.722378 15613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:33:58.722391 15613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37381/jwks.json
I20260927 00:33:58.727027 15613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:40273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:58.727888 15613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:58.729039 15613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:58.736831 15621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:58.736850 15618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:58.736889 15613 server_base.cc:1034] running on GCE node
W20260927 00:33:58.736841 15619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:58.737639 15613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:58.738696 15613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:58.739845 15613 hybrid_clock.cc:648] HybridClock initialized: now 1790469238739817 us; error 42 us; skew 500 ppm
I20260927 00:33:58.741472 15613 webserver.cc:460] Webserver started at http://127.10.11.254:33717/ using document root <none> and password file <none>
I20260927 00:33:58.742010 15613 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:58.742067 15613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:58.742286 15613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:58.744136 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "41bff05fb11643dea6a44687761cdb76"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "ab50e2433906fea39fb6226995b9116d"
server_key_iv: "2d9c70d1699e19ee6e6448674e355e5d"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.744668 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "41bff05fb11643dea6a44687761cdb76"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "ab50e2433906fea39fb6226995b9116d"
server_key_iv: "2d9c70d1699e19ee6e6448674e355e5d"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.748091 15613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:33:58.750278 15627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:58.751427 15613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:33:58.751551 15613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "41bff05fb11643dea6a44687761cdb76"
format_stamp: "Formatted at 2026-09-27 00:33:58 on dist-test-slave-6gxc"
server_key: "ab50e2433906fea39fb6226995b9116d"
server_key_iv: "2d9c70d1699e19ee6e6448674e355e5d"
server_key_version: "encryptionkey@0"
I20260927 00:33:58.751645 15613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:58.764837 15613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:58.765399 15613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:58.765573 15613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:58.772207 15613 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:40273
I20260927 00:33:58.772212 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:40273 every 8 connection(s)
I20260927 00:33:58.773048 15613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:58.775990 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:58.779068 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15613
I20260927 00:33:58.779206 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:58.779495 10287 external_mini_cluster.cc:1428] Setting key 817ac869132cd489b59c0843bf933b47
I20260927 00:33:58.781589 15680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76: Bootstrap starting.
I20260927 00:33:58.783664 15680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:58.784346 15680 log.cc:824] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:58.785989 15680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76: No bootstrap required, opened a new log
I20260927 00:33:58.788393 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.780791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.10.11.254:40273)
0927 00:33:58.781401 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.781423 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.781456 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:33:58.782145 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.782172 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.782548 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.782858 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:58.783273 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.784263 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.785181 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.785628 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.785791 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.786644 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:58.786649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:58.786670 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:58.787016 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:58.787066 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:58.787085 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:58.787192 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:58.787311 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:33:58.787557 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260927 00:33:58.788920 15680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } }
I20260927 00:33:58.789135 15680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:58.789240 15680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bff05fb11643dea6a44687761cdb76, State: Initialized, Role: FOLLOWER
I20260927 00:33:58.789691 15680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } }
I20260927 00:33:58.789835 15680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:58.789899 15680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:58.790019 15680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:58.790959 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:33:58.790939 15680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } }
I20260927 00:33:58.791247 15680 leader_election.cc:302] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41bff05fb11643dea6a44687761cdb76; no voters:
I20260927 00:33:58.791568 15680 leader_election.cc:288] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:58.791646 15685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:58.791810 15685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [term 1 LEADER]: Becoming Leader. State: Replica: 41bff05fb11643dea6a44687761cdb76, State: Running, Role: LEADER
I20260927 00:33:58.792078 15685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } }
I20260927 00:33:58.792707 15680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:58.793166 15687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bff05fb11643dea6a44687761cdb76. Latest consensus state: current_term: 1 leader_uuid: "41bff05fb11643dea6a44687761cdb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } } }
I20260927 00:33:58.793301 15687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:58.793586 15686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41bff05fb11643dea6a44687761cdb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bff05fb11643dea6a44687761cdb76" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 40273 } } }
I20260927 00:33:58.793701 15686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76 [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:58.793747 15691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:58.796443 15691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:58.801764 15691 catalog_manager.cc:1347] Generated new cluster ID: 13ed801215bb41f3bf0711cd5bb28b37
I20260927 00:33:58.801836 15691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:58.819921 15691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:58.821076 15691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:58.833117 15691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bff05fb11643dea6a44687761cdb76: Generated new TSK 0
I20260927 00:33:58.833877 15691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:33:58.847826 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:58.842989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.10.11.254:40273)
0927 00:33:58.843111 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:58.843116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:58.843284 (+ 168us) server_negotiation.cc:407] Connection header received
0927 00:33:58.843398 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:58.843401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:58.843453 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:58.843561 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:33:58.843757 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.844574 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.845388 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.845737 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.845838 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.846120 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:33:58.847490 (+ 1370us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:33:58.847706 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:33:58.847890 15683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 00:33:58.849576 15709 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:33:58.842896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:40273 (local address 127.0.0.1:51456)
0927 00:33:58.843205 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:58.843217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:58.843304 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:58.843565 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:58.843567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:58.843573 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:33:58.843663 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.843668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.844683 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.844686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.845288 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:58.845295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:58.845841 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:58.845844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:58.845927 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:58.845948 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:33:58.847650 (+ 1702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:33:58.849419 (+ 1769us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.11.254:40273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1590,"thread_start_us":115,"threads_started":1}
I20260927 00:33:58.850024 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15613
E20260927 00:33:58.858981 15710 webserver.cc:533] Webserver: sslize failed:
E20260927 00:33:58.859040 15710 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:33:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:33:58Z Disabled control of system clock
I20260927 00:33:58.897315 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42359
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42359
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/testchainca.cert with env {}
W20260927 00:33:59.006624 15714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:33:59.006852 15714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:33:59.006888 15714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:33:59.010792 15714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:33:59.010834 15714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:33:59.010849 15714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:33:59.010865 15714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:33:59.010879 15714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:33:59.010937 15714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:33:59.010957 15714 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:33:59.016033 15714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:33:59.016953 15714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:33:59.017858 15714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:33:59.026175 15720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:33:59.026185 15719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:59.026293 15714 server_base.cc:1034] running on GCE node
W20260927 00:33:59.026225 15722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:33:59.026984 15714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:33:59.027949 15714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:33:59.029127 15714 hybrid_clock.cc:648] HybridClock initialized: now 1790469239029091 us; error 45 us; skew 500 ppm
I20260927 00:33:59.034894 15714 webserver.cc:460] Webserver started at http://127.10.11.254:39241/ using document root <none> and password file <none>
I20260927 00:33:59.035398 15714 fs_manager.cc:359] Metadata directory not provided
I20260927 00:33:59.035454 15714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:33:59.035601 15714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:33:59.037447 15714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "f3ab429f3bff46ac858ccbfed09e039e"
format_stamp: "Formatted at 2026-09-27 00:33:59 on dist-test-slave-6gxc"
server_key: "432dd2856bf2f08b6f5b50c097c22cea"
server_key_iv: "55a11564bed646c73e6185319eccfeac"
server_key_version: "encryptionkey@0"
I20260927 00:33:59.037930 15714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "f3ab429f3bff46ac858ccbfed09e039e"
format_stamp: "Formatted at 2026-09-27 00:33:59 on dist-test-slave-6gxc"
server_key: "432dd2856bf2f08b6f5b50c097c22cea"
server_key_iv: "55a11564bed646c73e6185319eccfeac"
server_key_version: "encryptionkey@0"
I20260927 00:33:59.041281 15714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:33:59.043563 15728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:33:59.044713 15714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:33:59.044804 15714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "f3ab429f3bff46ac858ccbfed09e039e"
format_stamp: "Formatted at 2026-09-27 00:33:59 on dist-test-slave-6gxc"
server_key: "432dd2856bf2f08b6f5b50c097c22cea"
server_key_iv: "55a11564bed646c73e6185319eccfeac"
server_key_version: "encryptionkey@0"
I20260927 00:33:59.044926 15714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:33:59.056589 15714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:33:59.057152 15714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:33:59.057341 15714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:33:59.063922 15714 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42359
I20260927 00:33:59.064214 15780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42359 every 8 connection(s)
I20260927 00:33:59.064865 15714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:33:59.067456 15781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:33:59.072243 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15714
I20260927 00:33:59.072359 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:33:59.072399 15781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e: Bootstrap starting.
I20260927 00:33:59.072685 10287 external_mini_cluster.cc:1428] Setting key 6907f8af41d8daa145717aeabde806c0
I20260927 00:33:59.074779 15781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e: Neither blocks nor log segments found. Creating new log.
I20260927 00:33:59.075466 15781 log.cc:824] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:33:59.077095 15781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e: No bootstrap required, opened a new log
I20260927 00:33:59.080576 15781 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } }
I20260927 00:33:59.080840 15781 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:33:59.080950 15781 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ab429f3bff46ac858ccbfed09e039e, State: Initialized, Role: FOLLOWER
I20260927 00:33:59.081364 15781 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } }
I20260927 00:33:59.081511 15781 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:33:59.081562 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:59.073925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.10.11.254:42359)
0927 00:33:59.074521 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:59.074534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:59.074567 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:33:59.075247 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:59.075267 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:59.075678 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:59.075992 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:33:59.076381 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.077338 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.078265 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.078714 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.078904 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:59.079725 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:33:59.079730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:33:59.079741 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:33:59.080099 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:33:59.080150 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:33:59.080169 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:33:59.080280 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:33:59.080431 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:33:59.080665 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260927 00:33:59.081596 15781 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:33:59.081754 15781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:33:59.082666 15781 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } }
I20260927 00:33:59.083006 15781 leader_election.cc:302] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3ab429f3bff46ac858ccbfed09e039e; no voters:
I20260927 00:33:59.083302 15781 leader_election.cc:288] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:33:59.083408 15786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:33:59.083623 15786 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [term 1 LEADER]: Becoming Leader. State: Replica: f3ab429f3bff46ac858ccbfed09e039e, State: Running, Role: LEADER
I20260927 00:33:59.083938 15786 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } }
I20260927 00:33:59.084512 15781 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:33:59.084578 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:33:59.085067 15786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ab429f3bff46ac858ccbfed09e039e. Latest consensus state: current_term: 1 leader_uuid: "f3ab429f3bff46ac858ccbfed09e039e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } } }
I20260927 00:33:59.085186 15786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:59.085543 15791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:33:59.085774 15787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3ab429f3bff46ac858ccbfed09e039e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ab429f3bff46ac858ccbfed09e039e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42359 } } }
I20260927 00:33:59.085930 15787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e [sys.catalog]: This master's current role is: LEADER
I20260927 00:33:59.088536 15791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:33:59.093564 15791 catalog_manager.cc:1347] Generated new cluster ID: 9d3c2e0321f446edbc8133a3a3feb3d2
I20260927 00:33:59.093636 15791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:33:59.111577 15791 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:33:59.112422 15791 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:33:59.121637 15791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3ab429f3bff46ac858ccbfed09e039e: Generated new TSK 0
I20260927 00:33:59.122349 15791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:33:59.168143 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:33:59.130817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.10.11.254:42359)
0927 00:33:59.130946 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:33:59.130951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:33:59.131097 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:33:59.131211 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:33:59.131214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:33:59.131266 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:33:59.131373 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:33:59.131579 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.132402 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.133257 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.133596 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.133690 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:59.133981 (+ 291us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:33:59.167753 (+ 33772us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:33:59.168008 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:33:59.168212 15784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 00:33:59.168286 15810 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:33:59.130715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42359 (local address 127.0.0.1:33990)
0927 00:33:59.131015 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:33:59.131027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:33:59.131117 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:33:59.131374 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:33:59.131376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:33:59.131382 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:33:59.131485 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:59.131489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.132515 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.132518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:59.133151 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:33:59.133159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:33:59.133713 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:33:59.133716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:33:59.133811 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:33:59.133831 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:33:59.167949 (+ 34118us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:33:59.168095 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.11.254:42359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260927 00:33:59.168888 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15714
2026-09-27T00:33:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (299 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (115 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:33:59 dist-test-slave-6gxc krb5kdc[15822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:33:59 dist-test-slave-6gxc krb5kdc[15822](info): set up 2 sockets
Sep 27 00:33:59 dist-test-slave-6gxc krb5kdc[15822](info): commencing operation
krb5kdc: starting...
W20260927 00:34:00.426851 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:00 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:34:00.543066 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42763
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:00.649646 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:00.649933 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:00.650012 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:00.654915 15835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:00.655006 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:00.655049 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:00.655112 15835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:00.655153 15835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:00.660282 15835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:00.661254 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:00.662242 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:00.670424 15840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:00.670459 15841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:00.670463 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:00.670634 15835 server_base.cc:1034] running on GCE node
I20260927 00:34:00.671226 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:00.672199 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:00.673362 15835 hybrid_clock.cc:648] HybridClock initialized: now 1790469240673336 us; error 38 us; skew 500 ppm
Sep 27 00:34:00 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:00.676115 15835 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:34:00.676880 15835 webserver.cc:460] Webserver started at http://127.10.11.254:37603/ using document root <none> and password file <none>
I20260927 00:34:00.677342 15835 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:00.677399 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:00.677621 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:00.679514 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "ff20fdec6f3483ceb6fb7f32d2f8ad33"
server_key_iv: "4d3ffa51f31ebc6b2859e44790c07131"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.680004 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "ff20fdec6f3483ceb6fb7f32d2f8ad33"
server_key_iv: "4d3ffa51f31ebc6b2859e44790c07131"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.683391 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:00.685501 15850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:00.686748 15835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:34:00.686901 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "ff20fdec6f3483ceb6fb7f32d2f8ad33"
server_key_iv: "4d3ffa51f31ebc6b2859e44790c07131"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.686990 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:00.704005 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:00.707036 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:00.707211 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:00.713559 15835 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42763
I20260927 00:34:00.713624 15902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42763 every 8 connection(s)
I20260927 00:34:00.714447 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:00.717003 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:00.718232 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15835
I20260927 00:34:00.718362 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:00.718619 10287 external_mini_cluster.cc:1428] Setting key d50ad7c6451ea9e49cd15518f8d28719
I20260927 00:34:00.723455 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f: Bootstrap starting.
I20260927 00:34:00.725895 15903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:00.726735 15903 log.cc:824] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:34:00 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:00.728865 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f: No bootstrap required, opened a new log
I20260927 00:34:00.733582 15903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } }
I20260927 00:34:00.733772 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:00.719823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.10.11.254:42763)
0927 00:34:00.720223 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:00.720243 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:00.720322 (+ 79us) server_negotiation.cc:407] Connection header received
0927 00:34:00.721248 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:00.721270 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:00.721658 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:00.722031 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:00.722749 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.724117 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.724961 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.725603 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.725803 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:00.728524 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:00.728530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:00.728545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:00.728576 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:00.730710 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:00.731167 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:00.731174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:00.731182 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:00.731353 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:00.731674 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:00.731677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:00.731679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:00.732151 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:00.732364 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:00.732546 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:34:00.732845 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":100,"threads_started":1}
I20260927 00:34:00.733855 15903 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:00.733903 15903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fa9b158b784e42a0d5097bfa2c3f5f, State: Initialized, Role: FOLLOWER
I20260927 00:34:00.734362 15903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } }
I20260927 00:34:00.734526 15903 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:00.734601 15903 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:00.734705 15903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:00.735793 15903 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } }
I20260927 00:34:00.736137 15903 leader_election.cc:302] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7fa9b158b784e42a0d5097bfa2c3f5f; no voters:
I20260927 00:34:00.736443 15903 leader_election.cc:288] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:00.736522 15908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:00.736776 15908 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [term 1 LEADER]: Becoming Leader. State: Replica: b7fa9b158b784e42a0d5097bfa2c3f5f, State: Running, Role: LEADER
I20260927 00:34:00.737072 15908 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } }
I20260927 00:34:00.737612 15903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:00.738775 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7fa9b158b784e42a0d5097bfa2c3f5f. Latest consensus state: current_term: 1 leader_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } } }
I20260927 00:34:00.738773 15909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa9b158b784e42a0d5097bfa2c3f5f" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42763 } } }
I20260927 00:34:00.738925 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:00.738925 15909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:00.739358 15917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:00.742373 15917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:00.747503 15917 catalog_manager.cc:1347] Generated new cluster ID: 0e32b8271f2c4ffabad707aa7e4aee8f
I20260927 00:34:00.747556 15917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:00.776683 15917 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:00.777972 15917 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:00.784879 15917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7fa9b158b784e42a0d5097bfa2c3f5f: Generated new TSK 0
I20260927 00:34:00.785671 15917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:34:00.813490 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:00.918871 15929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:00.919092 15929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:00.919131 15929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:00.923223 15929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:00.923281 15929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:00.923363 15929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:00.928529 15929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:00.929441 15929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:00.930434 15929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:00.939419 15937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:00.939427 15935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:00.939496 15934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:00.939821 15929 server_base.cc:1034] running on GCE node
I20260927 00:34:00.940223 15929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:00.940819 15929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:00.942009 15929 hybrid_clock.cc:648] HybridClock initialized: now 1790469240941982 us; error 50 us; skew 500 ppm
Sep 27 00:34:00 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:00.944766 15929 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:34:00.945559 15929 webserver.cc:460] Webserver started at http://127.10.11.193:43285/ using document root <none> and password file <none>
I20260927 00:34:00.946089 15929 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:00.946233 15929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:00.946453 15929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:00.948348 15929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "ca8302afae1246319a13a6638bf06b98"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "f6c664890c15e53085a11e9028fe0c5e"
server_key_iv: "5c283cea5bb7ff7d0bd11b1dfb7b5d7c"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.948894 15929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "ca8302afae1246319a13a6638bf06b98"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "f6c664890c15e53085a11e9028fe0c5e"
server_key_iv: "5c283cea5bb7ff7d0bd11b1dfb7b5d7c"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.952348 15929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:34:00.954646 15944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:00.955823 15929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:00.955963 15929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "ca8302afae1246319a13a6638bf06b98"
format_stamp: "Formatted at 2026-09-27 00:34:00 on dist-test-slave-6gxc"
server_key: "f6c664890c15e53085a11e9028fe0c5e"
server_key_iv: "5c283cea5bb7ff7d0bd11b1dfb7b5d7c"
server_key_version: "encryptionkey@0"
I20260927 00:34:00.956069 15929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:00.969475 15929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:00.972599 15929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:00.972831 15929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:00.973389 15929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:00.974236 15929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:00.974311 15929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:00.974382 15929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:00.974433 15929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:00.982963 15929 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:39331
I20260927 00:34:00.982992 16057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:39331 every 8 connection(s)
I20260927 00:34:00.983801 15929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:00.988816 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 15929
I20260927 00:34:00.988937 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:00.989240 10287 external_mini_cluster.cc:1428] Setting key dcec4ea3263fcf1aaf8b34ba02d42674
Sep 27 00:34:00 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:00.998538 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:00.985484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:45409 (local address 127.10.11.254:42763)
0927 00:34:00.985610 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:00.985614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:00.986466 (+ 852us) server_negotiation.cc:407] Connection header received
0927 00:34:00.987832 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:00.987836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:00.987890 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:00.987975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:00.989275 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.989992 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.991651 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.991875 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.991958 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:00.994669 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:00.994676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:00.994682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:00.994714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:00.996405 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:00.996912 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:00.996918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:00.996921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:00.997011 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:00.997337 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:00.997344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:00.997348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:00.997581 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:00.997688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:00.998203 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 00:34:00.998326 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:34:00.999264 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:00.985865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42763 (local address 127.10.11.193:45409)
0927 00:34:00.986320 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:00.986360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:00.987628 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:00.988129 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:00.988135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:00.988461 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:00.989118 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:00.989136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.990155 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.990158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:00.991537 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:00.991550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:00.992021 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:00.992028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:00.992267 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:00.992897 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:00.992912 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:00.994506 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:00.996527 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:00.996531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:00.996542 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:00.996798 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:00.997104 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:00.997107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:00.997108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:00.997232 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:00.997681 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:00.997685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:00.997924 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 00:34:00.998068 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:34:00.998359 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260927 00:34:01.000428 16058 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42763
I20260927 00:34:01.000656 16058 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:01.001128 16058 heartbeater.cc:511] Master 127.10.11.254:42763 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:34:01.002738 15867 ts_manager.cc:194] Registered new tserver with Master: ca8302afae1246319a13a6638bf06b98 (127.10.11.193:39331)
I20260927 00:34:01.003906 15867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:45409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:34:01.016220 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:01.122885 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:01.123108 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:01.123148 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:01.127031 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:01.127076 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:01.127131 16063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:01.132046 16063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:01.132887 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:01.133729 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:01.143000 16068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:01.143011 16069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:01.143033 16071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:01.143087 16063 server_base.cc:1034] running on GCE node
I20260927 00:34:01.143574 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:01.144133 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:01.145318 16063 hybrid_clock.cc:648] HybridClock initialized: now 1790469241145307 us; error 30 us; skew 500 ppm
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.147961 16063 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:34:01.148716 16063 webserver.cc:460] Webserver started at http://127.10.11.194:38801/ using document root <none> and password file <none>
I20260927 00:34:01.149181 16063 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:01.149235 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:01.149384 16063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:01.151283 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "6f270bf2bd9c4623a8458c97ff58a2ae"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "af95fb46153c1bc808f545adddbaf02a"
server_key_iv: "d25b36b69269a23cbd98663f3374125a"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.151767 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "6f270bf2bd9c4623a8458c97ff58a2ae"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "af95fb46153c1bc808f545adddbaf02a"
server_key_iv: "d25b36b69269a23cbd98663f3374125a"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.155097 16063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:01.157230 16078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.158417 16063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:01.158505 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "6f270bf2bd9c4623a8458c97ff58a2ae"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "af95fb46153c1bc808f545adddbaf02a"
server_key_iv: "d25b36b69269a23cbd98663f3374125a"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.158579 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:01.178829 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:01.182386 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:01.182577 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:01.183125 16063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:01.183885 16063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:01.183931 16063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.184006 16063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:01.184046 16063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.192512 16063 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:34529
I20260927 00:34:01.192602 16191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:34529 every 8 connection(s)
I20260927 00:34:01.193367 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:01.201555 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16063
I20260927 00:34:01.201674 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:01.201978 10287 external_mini_cluster.cc:1428] Setting key 85bfd16c3f1631e222df6f87f790da00
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:01.206858 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.195073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:59671 (local address 127.10.11.254:42763)
0927 00:34:01.195193 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.195196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.195933 (+ 737us) server_negotiation.cc:407] Connection header received
0927 00:34:01.197328 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.197332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.197396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.197495 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.198725 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.199248 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.200363 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.200584 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.200703 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.203121 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:01.203125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:01.203128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:01.203158 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:01.204554 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.205195 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.205198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.205199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.205276 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.205651 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.205653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.205654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.205810 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:01.205883 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:01.206475 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 00:34:01.206617 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:34:01.207402 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.195380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42763 (local address 127.10.11.194:59671)
0927 00:34:01.195792 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:01.195832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:01.197105 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:01.197661 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:01.197667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:01.198002 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:01.198586 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:01.198600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.199380 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.199384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:01.200238 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:01.200247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.200736 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.200740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:01.200913 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.201407 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:01.201422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:01.202959 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:01.204696 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:01.204704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:01.204717 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:01.205086 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:01.205372 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:01.205377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:01.205380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:01.205557 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:01.205916 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:01.205923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:01.206246 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 00:34:01.206399 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:34:01.206619 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260927 00:34:01.208492 16192 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42763
I20260927 00:34:01.208738 16192 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:01.209319 16192 heartbeater.cc:511] Master 127.10.11.254:42763 requested a full tablet report, sending...
I20260927 00:34:01.210397 15867 ts_manager.cc:194] Registered new tserver with Master: 6f270bf2bd9c4623a8458c97ff58a2ae (127.10.11.194:34529)
I20260927 00:34:01.210958 15867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:59671
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:34:01.228435 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:01.336388 16197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:01.336629 16197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:01.336669 16197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:01.340931 16197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:01.340986 16197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:01.341081 16197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:01.346244 16197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:01.347183 16197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:01.348146 16197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:01.357510 16197 server_base.cc:1034] running on GCE node
W20260927 00:34:01.357425 16203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:01.357461 16202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:01.357438 16205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:01.358001 16197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:01.358659 16197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:01.359836 16197 hybrid_clock.cc:648] HybridClock initialized: now 1790469241359803 us; error 46 us; skew 500 ppm
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.362641 16197 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:34:01.363539 16197 webserver.cc:460] Webserver started at http://127.10.11.195:33663/ using document root <none> and password file <none>
I20260927 00:34:01.364076 16197 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:01.364164 16197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:01.364362 16197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:01.366324 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "87ba7d583fd443a991247ca23e76e76e"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "53cd208a1dfb62fae81747bec05a26ec"
server_key_iv: "eac88a40e6a2fb54009f0d168241436b"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.366878 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "87ba7d583fd443a991247ca23e76e76e"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "53cd208a1dfb62fae81747bec05a26ec"
server_key_iv: "eac88a40e6a2fb54009f0d168241436b"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.370332 16197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:34:01.372611 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.373881 16197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:01.374013 16197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "87ba7d583fd443a991247ca23e76e76e"
format_stamp: "Formatted at 2026-09-27 00:34:01 on dist-test-slave-6gxc"
server_key: "53cd208a1dfb62fae81747bec05a26ec"
server_key_iv: "eac88a40e6a2fb54009f0d168241436b"
server_key_version: "encryptionkey@0"
I20260927 00:34:01.374148 16197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:01.395654 16197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:01.398567 16197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:01.398764 16197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:01.399313 16197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:01.400089 16197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:01.400168 16197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.400261 16197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:01.400312 16197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:01.409303 16197 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:34915
I20260927 00:34:01.409338 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:34915 every 8 connection(s)
I20260927 00:34:01.410218 16197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:01.414043 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16197
I20260927 00:34:01.414175 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:01.414420 10287 external_mini_cluster.cc:1428] Setting key 79e70aa037d148d0c23d6d94ea700cc6
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:01.424075 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.412007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:40121 (local address 127.10.11.254:42763)
0927 00:34:01.412153 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.412157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.413679 (+ 1522us) server_negotiation.cc:407] Connection header received
0927 00:34:01.415248 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.415251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.415303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.415410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.416522 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.417037 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.418158 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.418375 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.418495 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.420795 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:01.420799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:01.420803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:01.420837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:01.422084 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.422615 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.422620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.422622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.422700 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.423034 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.423040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.423041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.423220 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:01.423338 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:01.423792 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 00:34:01.423911 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:34:01.424762 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.412305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42763 (local address 127.10.11.195:40121)
0927 00:34:01.413552 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:01.413591 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:01.415050 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:01.415580 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:01.415587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:01.415926 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:01.416380 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:01.416393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.417160 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.417164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:01.418013 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:01.418022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.418517 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.418521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:01.418692 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.419260 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:01.419275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:01.420640 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:01.422246 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:01.422251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:01.422264 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:01.422509 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:01.422798 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:01.422801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:01.422803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:01.422931 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:01.423342 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:01.423350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:01.423576 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:34:01.423753 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:34:01.424037 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":179,"threads_started":1}
I20260927 00:34:01.425752 16326 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42763
I20260927 00:34:01.425951 16326 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:01.426403 16326 heartbeater.cc:511] Master 127.10.11.254:42763 requested a full tablet report, sending...
I20260927 00:34:01.427381 15867 ts_manager.cc:194] Registered new tserver with Master: 87ba7d583fd443a991247ca23e76e76e (127.10.11.195:34915)
I20260927 00:34:01.427927 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:34:01.428095 15867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:40121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/test-admin.keytab.
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.443259 10287 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:34:01.445757 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:01.444437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.10.11.193:39331)
0927 00:34:01.444760 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.444769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.444790 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:34:01.444840 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.444844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.444996 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.445108 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.445554 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46444: BlockingRecv error: recv got EOF from 127.0.0.1:46444 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":142,"threads_started":1}
W20260927 00:34:01.445866 10287 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.447048 10287 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.447942 10287 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:34:01.457343 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.448692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.10.11.193:39331)
0927 00:34:01.448867 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.448870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.448897 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:34:01.449004 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.449006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.449041 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.449105 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.449555 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.450381 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.451316 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.451656 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.451740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.454021 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:01.454030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:01.454040 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:01.454062 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:01.455648 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.456072 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.456076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.456078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.456154 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.456402 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.456405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.456406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.456615 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:01.456770 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:01.457019 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:34:01.457183 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 00:34:01.461275 10287 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc' of type FILE
W20260927 00:34:01.461294 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:01.460155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.10.11.193:39331)
0927 00:34:01.460315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.460319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.460495 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:34:01.460570 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.460573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.460616 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.460683 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.461153 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46456: BlockingRecv error: recv got EOF from 127.0.0.1:46456 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.462378 10287 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.463361 10287 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:34:01.471372 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.464104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.10.11.193:39331)
0927 00:34:01.464224 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.464228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.464243 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:34:01.464347 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.464349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.464380 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.464443 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.464883 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.465360 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.466063 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.466330 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.466407 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.468542 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:01.468545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:01.468547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:01.468570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:01.469906 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.470360 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.470362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.470363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.470436 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.470642 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.470643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.470645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.470791 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:01.470880 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:01.471092 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:34:01.471170 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:34:01.473789 16333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:01.472926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.10.11.193:39331)
0927 00:34:01.473043 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.473046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.473085 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:34:01.473177 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.473179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.473209 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.473271 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.473720 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46484: BlockingRecv error: recv got EOF from 127.0.0.1:46484 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:34:01.473896 10287 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469213433099-10287-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.474884 10287 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:01.475850 10287 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:34:01.483856 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:01.476596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.10.11.193:39331)
0927 00:34:01.476742 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:01.476745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:01.476758 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:01.476824 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:01.476826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:01.476855 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:01.476916 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:01.477359 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.478004 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.478880 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:01.479078 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:01.479158 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:01.481266 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:01.481269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:01.481271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:01.481290 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:01.482519 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.482873 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.482875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.482876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.482938 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:01.483243 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:01.483246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:01.483247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:01.483399 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:01.483487 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:01.483676 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:34:01.483751 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:34:01.484766 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15929
I20260927 00:34:01.490666 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16063
I20260927 00:34:01.495874 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16197
I20260927 00:34:01.501001 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 15835
2026-09-27T00:34:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2100 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[16346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[16346](info): set up 2 sockets
Sep 27 00:34:01 dist-test-slave-6gxc krb5kdc[16346](info): commencing operation
krb5kdc: starting...
W20260927 00:34:02.539381 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:02 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab.
Entry for principal oryx/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab.
I20260927 00:34:02.655016 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:35645
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab
--principal=oryx/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:35645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:02.758881 16359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:02.759187 16359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:02.759249 16359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:02.763206 16359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:02.763252 16359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:02.763271 16359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:02.763288 16359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:02.763350 16359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:02.768661 16359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:35645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab
--principal=oryx/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:02.769683 16359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:02.770619 16359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:02.778364 16359 server_base.cc:1034] running on GCE node
W20260927 00:34:02.778327 16364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:02.778319 16367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:02.778292 16365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:02.779152 16359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:02.780185 16359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:02.781380 16359 hybrid_clock.cc:648] HybridClock initialized: now 1790469242781355 us; error 45 us; skew 500 ppm
Sep 27 00:34:02 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:02.783982 16359 init.cc:377] Logged in from keytab as oryx/127.10.11.254@KRBTEST.COM (short username oryx)
I20260927 00:34:02.784817 16359 webserver.cc:460] Webserver started at http://127.10.11.254:38715/ using document root <none> and password file <none>
I20260927 00:34:02.785284 16359 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:02.785349 16359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:02.785498 16359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:02.787434 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "c76f503a46ff4540985caceb448ce3e7"
format_stamp: "Formatted at 2026-09-27 00:34:02 on dist-test-slave-6gxc"
server_key: "d35563e6b9c1e3210d8bdd4123b82c95"
server_key_iv: "43a4efeec1666d95a112867d4f427132"
server_key_version: "encryptionkey@0"
I20260927 00:34:02.787930 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "c76f503a46ff4540985caceb448ce3e7"
format_stamp: "Formatted at 2026-09-27 00:34:02 on dist-test-slave-6gxc"
server_key: "d35563e6b9c1e3210d8bdd4123b82c95"
server_key_iv: "43a4efeec1666d95a112867d4f427132"
server_key_version: "encryptionkey@0"
I20260927 00:34:02.791250 16359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:02.793344 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:02.794523 16359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:34:02.794637 16359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "c76f503a46ff4540985caceb448ce3e7"
format_stamp: "Formatted at 2026-09-27 00:34:02 on dist-test-slave-6gxc"
server_key: "d35563e6b9c1e3210d8bdd4123b82c95"
server_key_iv: "43a4efeec1666d95a112867d4f427132"
server_key_version: "encryptionkey@0"
I20260927 00:34:02.794731 16359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:02.808310 16359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:02.810846 16359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:02.811033 16359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:02.817543 16359 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:35645
I20260927 00:34:02.817818 16426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:35645 every 8 connection(s)
I20260927 00:34:02.818485 16359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:02.819916 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16359
I20260927 00:34:02.820044 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:02.820329 10287 external_mini_cluster.cc:1428] Setting key f97f49cc93ebc90b27a1f76b099206bf
I20260927 00:34:02.821455 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:02.828028 16427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Bootstrap starting.
Sep 27 00:34:02 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:02.830287 16427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:02.831195 16427 log.cc:824] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:02.833320 16427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: No bootstrap required, opened a new log
I20260927 00:34:02.835196 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:02.821608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.10.11.254:35645)
0927 00:34:02.822047 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:02.822065 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:02.822133 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:34:02.822975 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:02.822997 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:02.823392 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:02.823737 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:02.824336 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:02.825801 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:02.826692 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:02.827391 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:02.827565 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:02.830052 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:02.830058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:02.830072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:02.830122 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:02.832190 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:02.832620 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:02.832625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:02.832631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:02.832758 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:02.833066 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:02.833069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:02.833071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:02.833548 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:02.833731 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:02.834006 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:34:02.834318 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260927 00:34:02.837118 16427 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } }
I20260927 00:34:02.837316 16427 raft_consensus.cc:374] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:02.837352 16427 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76f503a46ff4540985caceb448ce3e7, State: Initialized, Role: FOLLOWER
I20260927 00:34:02.837682 16427 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } }
I20260927 00:34:02.837796 16427 raft_consensus.cc:388] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:02.837833 16427 raft_consensus.cc:482] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:02.837893 16427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:02.838712 16427 raft_consensus.cc:504] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } }
I20260927 00:34:02.839004 16427 leader_election.cc:302] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c76f503a46ff4540985caceb448ce3e7; no voters:
I20260927 00:34:02.839231 16427 leader_election.cc:288] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:02.839347 16432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:02.839577 16432 raft_consensus.cc:686] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 LEADER]: Becoming Leader. State: Replica: c76f503a46ff4540985caceb448ce3e7, State: Running, Role: LEADER
I20260927 00:34:02.839897 16432 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } }
I20260927 00:34:02.840456 16427 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:02.841446 16433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } }
I20260927 00:34:02.841579 16433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:02.841852 16434 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76f503a46ff4540985caceb448ce3e7. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } }
I20260927 00:34:02.841939 16434 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:02.841926 16441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:02.845062 16441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:02.850090 16441 catalog_manager.cc:1347] Generated new cluster ID: 72017ef12c144e48a44494858c9392df
I20260927 00:34:02.850205 16441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:02.872385 16441 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:02.873351 16441 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:02.881142 16441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Generated new TSK 0
I20260927 00:34:02.881830 16441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
I20260927 00:34:02.917266 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:03.042642 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:03.042915 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:03.042986 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:03.046942 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:03.046994 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:03.047077 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:03.052002 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:03.052944 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:03.053885 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:03.062908 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.062898 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.062898 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.063127 16453 server_base.cc:1034] running on GCE node
I20260927 00:34:03.063539 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:03.064095 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:03.065253 16453 hybrid_clock.cc:648] HybridClock initialized: now 1790469243065242 us; error 25 us; skew 500 ppm
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:03.067855 16453 init.cc:377] Logged in from keytab as oryx/127.10.11.193@KRBTEST.COM (short username oryx)
I20260927 00:34:03.068660 16453 webserver.cc:460] Webserver started at http://127.10.11.193:35175/ using document root <none> and password file <none>
I20260927 00:34:03.069164 16453 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.069222 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.069414 16453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:03.071260 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "b83e20ead07c49c18bb6b753d04d350e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "06d2f7bb5fa7631cf4c1bfedccd4cd40"
server_key_iv: "d42e8ce41ad2c863a57c0b2bb78e2783"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.071806 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "b83e20ead07c49c18bb6b753d04d350e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "06d2f7bb5fa7631cf4c1bfedccd4cd40"
server_key_iv: "d42e8ce41ad2c863a57c0b2bb78e2783"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.075189 16453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:03.077468 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.078640 16453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:03.078763 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "b83e20ead07c49c18bb6b753d04d350e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "06d2f7bb5fa7631cf4c1bfedccd4cd40"
server_key_iv: "d42e8ce41ad2c863a57c0b2bb78e2783"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.078864 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.100490 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:03.103286 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:03.103482 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:03.104013 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:03.104763 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:03.104832 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.104897 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:03.104943 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.113933 16453 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:38955
I20260927 00:34:03.114002 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:38955 every 8 connection(s)
I20260927 00:34:03.114902 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:03.123497 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16453
I20260927 00:34:03.123626 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:03.123929 10287 external_mini_cluster.cc:1428] Setting key 2cf8dd91758d4936deeb95c7e6fee76a
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:03.129065 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.116924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:36783 (local address 127.10.11.254:35645)
0927 00:34:03.117069 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.117074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.117921 (+ 847us) server_negotiation.cc:407] Connection header received
0927 00:34:03.119306 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.119308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.119355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.119427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.120521 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.121049 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.122257 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.122491 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.122573 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.125152 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.125158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.125165 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.125188 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.126960 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.127428 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.127433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.127436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.127543 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.127862 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.127867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.127871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.128104 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.128219 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.128659 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.128818 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:34:03.129549 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.117330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:36783)
0927 00:34:03.117778 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.117819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.119045 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.119577 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.119583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.119943 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.120390 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.120402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.121165 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.121168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.122147 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.122160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.122640 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.122650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.122817 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.123330 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.123353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.124990 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.127065 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.127069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.127079 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.127326 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.127636 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.127638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.127639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.127772 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.128218 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.128222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.128429 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.128574 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.128796 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260927 00:34:03.130836 16582 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
I20260927 00:34:03.131026 16582 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:03.131417 16582 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:03.132849 16391 ts_manager.cc:194] Registered new tserver with Master: b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
WARNING: no policy specified for oryx/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:34:03.133976 16391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:36783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
I20260927 00:34:03.149375 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:03.256049 16587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:03.256263 16587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:03.256302 16587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:03.260273 16587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:03.260321 16587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:03.260416 16587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:03.265306 16587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:03.266243 16587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:03.267141 16587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:03.276122 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.276227 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.276172 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.276309 16587 server_base.cc:1034] running on GCE node
I20260927 00:34:03.276734 16587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:03.277344 16587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:03.278534 16587 hybrid_clock.cc:648] HybridClock initialized: now 1790469243278500 us; error 46 us; skew 500 ppm
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:03.281200 16587 init.cc:377] Logged in from keytab as oryx/127.10.11.194@KRBTEST.COM (short username oryx)
I20260927 00:34:03.281961 16587 webserver.cc:460] Webserver started at http://127.10.11.194:35457/ using document root <none> and password file <none>
I20260927 00:34:03.282558 16587 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.282617 16587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.282766 16587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:03.284682 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "d4bf4552f23f4064a5ecbab3836a5fe3"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "0b897b9c8ac07112106da16a32577665"
server_key_iv: "73786dc5e4929d8eec932fe4683006e9"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.285161 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "d4bf4552f23f4064a5ecbab3836a5fe3"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "0b897b9c8ac07112106da16a32577665"
server_key_iv: "73786dc5e4929d8eec932fe4683006e9"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.288575 16587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:34:03.290728 16602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.291859 16587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:03.291980 16587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "d4bf4552f23f4064a5ecbab3836a5fe3"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "0b897b9c8ac07112106da16a32577665"
server_key_iv: "73786dc5e4929d8eec932fe4683006e9"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.292078 16587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.318432 16587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:03.321611 16587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:03.321818 16587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:03.322422 16587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:03.323248 16587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:03.323292 16587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.323359 16587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:03.323401 16587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.331650 16587 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:35729
I20260927 00:34:03.331672 16715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:35729 every 8 connection(s)
I20260927 00:34:03.332566 16587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:03.334609 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16587
I20260927 00:34:03.334734 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:03.335043 10287 external_mini_cluster.cc:1428] Setting key 21a351b6a0ea5b383a478b40187d5c4f
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.195@KRBTEST.COM" created.
I20260927 00:34:03.347234 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.334533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:34761 (local address 127.10.11.254:35645)
0927 00:34:03.334670 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.334676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.335649 (+ 973us) server_negotiation.cc:407] Connection header received
0927 00:34:03.337021 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.337027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.337085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.337174 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.338350 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.339177 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.340256 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.340644 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.340770 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.343590 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.343593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.343595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.343615 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.345059 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.345566 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.345572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.345581 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.345667 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.346022 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.346027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.346028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.346351 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.346486 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.346963 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.347072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:34:03.347966 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.335002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:34761)
0927 00:34:03.335498 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.335549 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.336815 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.337319 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.337326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.337655 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.338209 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.338225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.339296 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.339298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.340140 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.340149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.340779 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.340781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.340967 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.341644 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.341659 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.343433 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.345159 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.345163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.345173 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.345436 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.345781 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.345783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.345786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.345924 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.346484 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.346491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.346706 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.346923 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.347208 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":149,"threads_started":1}
I20260927 00:34:03.349134 16716 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
I20260927 00:34:03.349381 16716 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:03.349828 16716 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:03.350883 16391 ts_manager.cc:194] Registered new tserver with Master: d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729)
I20260927 00:34:03.351492 16391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:34761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
I20260927 00:34:03.362556 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:03.468859 16721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:03.469069 16721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:03.469133 16721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:03.473107 16721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:03.473155 16721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:03.473244 16721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:03.478281 16721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:03.479136 16721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:03.480016 16721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:03.488994 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.489022 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.489054 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.489220 16721 server_base.cc:1034] running on GCE node
I20260927 00:34:03.489593 16721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:03.490197 16721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:03.491370 16721 hybrid_clock.cc:648] HybridClock initialized: now 1790469243491361 us; error 28 us; skew 500 ppm
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:03.493844 16721 init.cc:377] Logged in from keytab as oryx/127.10.11.195@KRBTEST.COM (short username oryx)
I20260927 00:34:03.494673 16721 webserver.cc:460] Webserver started at http://127.10.11.195:44107/ using document root <none> and password file <none>
I20260927 00:34:03.495177 16721 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.495262 16721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.495478 16721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:03.497402 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "d294934e21a446bd8c8a872ca3266d0c"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "3a2c7c8e7abec7b324fa389cb54b2c7d"
server_key_iv: "d6799b85c96c41a99adf373723f1d7b6"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.498008 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "d294934e21a446bd8c8a872ca3266d0c"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "3a2c7c8e7abec7b324fa389cb54b2c7d"
server_key_iv: "d6799b85c96c41a99adf373723f1d7b6"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.501617 16721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:34:03.504130 16736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.505400 16721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:34:03.505646 16721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "d294934e21a446bd8c8a872ca3266d0c"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "3a2c7c8e7abec7b324fa389cb54b2c7d"
server_key_iv: "d6799b85c96c41a99adf373723f1d7b6"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.505759 16721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.527247 16721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:03.530246 16721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:03.530427 16721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:03.530982 16721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:03.531723 16721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:03.531801 16721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.531889 16721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:03.531939 16721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.541054 16721 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:35921
I20260927 00:34:03.541065 16849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:35921 every 8 connection(s)
I20260927 00:34:03.541996 16721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:03.547896 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16721
I20260927 00:34:03.548025 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:03.548298 10287 external_mini_cluster.cc:1428] Setting key 100656a45094ed990ed012b69f610657
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:03.555176 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.543703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34887 (local address 127.10.11.254:35645)
0927 00:34:03.543846 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.543850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.544592 (+ 742us) server_negotiation.cc:407] Connection header received
0927 00:34:03.545922 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.545925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.545983 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.546089 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.547187 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.547719 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.549145 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.549481 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.549611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.552099 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.552102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.552104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.552127 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.553373 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.553828 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.553830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.553832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.553902 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.554302 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.554305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.554306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.554463 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.554539 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.554977 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.555065 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:34:03.555886 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.543968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:34887)
0927 00:34:03.544433 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.544477 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.545681 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.546272 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.546279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.546605 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.547038 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.547050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.547841 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.547846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.549022 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.549032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.549646 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.549650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.549882 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.550504 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.550519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.551937 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.553483 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.553486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.553496 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.553726 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.554008 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.554010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.554012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.554211 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.554558 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.554562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.554776 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.554916 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.555133 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":112,"threads_started":1}
I20260927 00:34:03.556919 16850 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
I20260927 00:34:03.557111 16850 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:03.557489 16850 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:03.558405 16391 ts_manager.cc:194] Registered new tserver with Master: d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:03.558892 16391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:34887
I20260927 00:34:03.562017 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.11.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.253.keytab.
Entry for principal oryx/127.10.11.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.253.keytab.
I20260927 00:34:03.588855 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:33761
--webserver_interface=127.10.11.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.253.keytab
--principal=oryx/127.10.11.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:35645,127.10.11.253:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:03.694486 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:03.694728 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:03.694794 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:03.699066 16855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:03.699134 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:03.699172 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:03.699210 16855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:03.699245 16855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:03.704218 16855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:35645,127.10.11.253:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.253.keytab
--principal=oryx/127.10.11.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.11.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
W20260927 00:34:03.704440 16855 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.11.254:35645,127.10.11.253:33761'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:34:03.705200 16855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:03.706132 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:03.714047 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.714054 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.714027 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.714208 16855 server_base.cc:1034] running on GCE node
I20260927 00:34:03.714848 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:03.715838 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:03.717023 16855 hybrid_clock.cc:648] HybridClock initialized: now 1790469243717001 us; error 38 us; skew 500 ppm
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:03.719609 16855 init.cc:377] Logged in from keytab as oryx/127.10.11.253@KRBTEST.COM (short username oryx)
I20260927 00:34:03.720352 16855 webserver.cc:460] Webserver started at http://127.10.11.253:46233/ using document root <none> and password file <none>
I20260927 00:34:03.720860 16855 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.720953 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.721176 16855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:03.723069 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/instance:
uuid: "fa5126e3120f4e01bf7835377955f83e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "fe916697841c202dba988c24a711d934"
server_key_iv: "538bd1a3487db0dcf771bc1905572d8a"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.723626 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal/instance:
uuid: "fa5126e3120f4e01bf7835377955f83e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "fe916697841c202dba988c24a711d934"
server_key_iv: "538bd1a3487db0dcf771bc1905572d8a"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.726961 16855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:03.729117 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.730415 16855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:03.730545 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "fa5126e3120f4e01bf7835377955f83e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "fe916697841c202dba988c24a711d934"
server_key_iv: "538bd1a3487db0dcf771bc1905572d8a"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.730648 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.743317 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:03.745949 16855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:03.746084 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:03.752588 16855 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.253:33761
I20260927 00:34:03.752851 16922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.253:33761 every 8 connection(s)
I20260927 00:34:03.753468 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
I20260927 00:34:03.753738 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16855
I20260927 00:34:03.756588 16923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:03.760831 16923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.253@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:03.768800 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.755217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.10.11.253:33761)
0927 00:34:03.755611 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.755627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.755666 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:34:03.756463 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.756484 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.756868 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.757221 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.757843 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.759325 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.760189 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.760831 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.761003 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.763913 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.763923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.763932 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.763957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.765844 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.766419 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.766425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.766430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.766553 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.766855 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.766860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.766865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.767372 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.767580 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.767812 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.768086 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260927 00:34:03.772078 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.762694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.10.11.254:35645)
0927 00:34:03.762867 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.762872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.763124 (+ 252us) server_negotiation.cc:407] Connection header received
0927 00:34:03.763217 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.763220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.763270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.763361 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.764056 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.764775 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.766127 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.766484 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.766562 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.768762 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.768765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.768767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.768796 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.770333 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.770824 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.770828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.770829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.770906 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.771186 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.771190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.771191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.771389 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.771495 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.771820 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.771923 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:34:03.772101 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.762605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57358)
0927 00:34:03.762974 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.763019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.763134 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.763476 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.763483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.763810 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.763927 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.763933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.764924 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.764927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.765984 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.765991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.766590 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.766596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.766712 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.767139 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.767153 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.768618 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.770441 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.770451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.770456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.770737 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.770997 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.771002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.771004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.771090 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.771493 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.771499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.771607 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.771738 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.771920 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260927 00:34:03.773182 16923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.253@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:03.780777 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.773606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44492)
0927 00:34:03.773691 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.773706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.773780 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.773983 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.773985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.774155 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.774240 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.774246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.775398 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.775400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.775956 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.775969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.776421 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.776424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.776495 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.776958 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.776971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.778220 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.779627 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.779629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.779631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.779812 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.780038 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.780040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.780041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.780111 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.780400 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.780402 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.780473 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.780551 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.780640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 00:34:03.780799 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.773682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.10.11.253:33761)
0927 00:34:03.773804 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.773807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.773825 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:34:03.773852 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.773854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.773886 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.773960 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.774378 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.775264 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.776091 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.776312 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.776412 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.778338 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.778341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.778343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.778362 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.779533 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.779892 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.779894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.779895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.779964 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.780182 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.780184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.780185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.780327 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.780394 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.780602 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.780685 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:34:03.783793 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Bootstrap starting.
I20260927 00:34:03.785846 16923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:03.786691 16923 log.cc:824] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:03.788333 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: No bootstrap required, opened a new log
I20260927 00:34:03.791255 16923 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } }
I20260927 00:34:03.791435 16923 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:03.791469 16923 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5126e3120f4e01bf7835377955f83e, State: Initialized, Role: FOLLOWER
I20260927 00:34:03.791853 16923 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } }
I20260927 00:34:03.792237 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } }
I20260927 00:34:03.792347 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:03.792986 16923 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:34:03.797559 16940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:34:03.797641 16940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:34:03.799860 16855 master_runner.cc:428] Detected that this master fa5126e3120f4e01bf7835377955f83e is joining an existing cluster
I20260927 00:34:03.799919 16855 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.11.253:33761
I20260927 00:34:03.809823 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.801998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44498)
0927 00:34:03.802388 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.802403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.802500 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.802720 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.802723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.802919 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.803007 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.803012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.803730 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.803734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.804295 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.804302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.804765 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.804769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.804851 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.805336 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.805355 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.805758 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.807862 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.807865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.807868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.808419 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.808824 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.808828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.808830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.808921 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.809328 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.809331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.809417 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.809533 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.809651 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":103,"threads_started":1}
I20260927 00:34:03.810151 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.802372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.10.11.253:33761)
0927 00:34:03.802453 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.802458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.802475 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:34:03.802593 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.802596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.802640 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.802709 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.803114 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.803620 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.804411 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.804648 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.804764 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.805879 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.805883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.805885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.805905 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.807753 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.808579 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.808584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.808588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.808673 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.809023 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.809027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.809029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.809234 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.809326 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.809647 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.809753 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":34}
I20260927 00:34:03.810757 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.802030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.10.11.254:35645)
0927 00:34:03.802177 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.802182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.802513 (+ 331us) server_negotiation.cc:407] Connection header received
0927 00:34:03.802681 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.802684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.802730 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.802801 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.803412 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.804208 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.805366 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.805722 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.805843 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.807110 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.807116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.807120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.807150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.808711 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.809257 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.809263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.809268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.809351 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.809646 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.809649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.809652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.809851 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.810137 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.810504 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.810615 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:34:03.811120 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.802130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57360)
0927 00:34:03.802431 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.802445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.802595 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.802906 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.802908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.803121 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.803281 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.803288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.804344 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.804347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.805258 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.805268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.805843 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.805847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.805938 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.806491 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.806506 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.806922 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.808845 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.808847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.808849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.809147 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.809452 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.809454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.809455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.809545 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.810216 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.810221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.810322 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.810897 (+ 575us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.811009 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":27,"thread_start_us":95,"threads_started":1}
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.254@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:03.824215 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.814847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.10.11.253:33761)
0927 00:34:03.814964 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.814968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.815302 (+ 334us) server_negotiation.cc:407] Connection header received
0927 00:34:03.815442 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.815444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.815482 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.815546 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.816217 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.816652 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.817870 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.818241 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.818335 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.820812 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.820816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.820818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.820841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.822489 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.823010 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.823013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.823015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.823090 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.823383 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.823386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.823388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.823556 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.823614 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.823994 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.824086 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:34:03.824862 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.814807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44506)
0927 00:34:03.815161 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.815218 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.815341 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.815597 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.815603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.815987 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.816107 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.816114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.816757 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.816761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.817717 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.817726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.818435 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.818439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.818539 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.819031 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.819045 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.820671 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.822603 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.822611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.822617 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.822899 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.823197 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.823200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.823202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.823283 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.823685 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.823690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.823802 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.824514 (+ 712us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.824689 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260927 00:34:03.826355 16391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.11.253:33761: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fa5126e3120f4e01bf7835377955f83e"
member_type: NON_VOTER
last_known_addr {
host: "127.10.11.253"
port: 33761
}
attrs {
promote: true
}
}
dest_uuid: "c76f503a46ff4540985caceb448ce3e7"
cas_config_opid_index: -1
I20260927 00:34:03.826951 16391 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } }
I20260927 00:34:03.828500 16950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa5126e3120f4e01bf7835377955f83e (127.10.11.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:03.829284 16897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:03.830376 16950 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.11.253:33761
I20260927 00:34:03.830634 16897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:34:03.830751 16855 master.cc:537] Master@127.10.11.253:33761 shutting down...
I20260927 00:34:03.831266 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:03.831387 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:03.831547 16951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:34:03.834007 16377 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:34:03.835240 16855 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:03.835628 16855 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:03.835927 16855 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: stopping tablet replica
W20260927 00:34:03.838688 16377 consensus_peers.cc:487] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 -> Peer fa5126e3120f4e01bf7835377955f83e (127.10.11.253:33761): Couldn't send request to peer fa5126e3120f4e01bf7835377955f83e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.10.11.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.252@KRBTEST.COM" created.
I20260927 00:34:03.851295 16855 master.cc:552] Master@127.10.11.253:33761 shutdown complete.
I20260927 00:34:03.851347 16855 master_runner.cc:305] Clearing existing system tablet
I20260927 00:34:03.851608 16855 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.851661 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.853113 16855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:34:03.854027 16956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.854314 16855 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.854485 16855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:03.854518 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "fa5126e3120f4e01bf7835377955f83e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "fe916697841c202dba988c24a711d934"
server_key_iv: "538bd1a3487db0dcf771bc1905572d8a"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.252.keytab.
Entry for principal oryx/127.10.11.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.252.keytab.
I20260927 00:34:03.856604 16855 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:03.857141 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:34887
--webserver_interface=127.10.11.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.252.keytab
--principal=oryx/127.10.11.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:34:03.858791 16855 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:34:03.858875 16855 log.cc:1196] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:34:03.859102 16855 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Deleting consensus metadata
I20260927 00:34:03.859274 16855 master_runner.cc:315] Copying system tablet from 127.10.11.254:35645
I20260927 00:34:03.860320 16855 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: tablet copy: Beginning tablet copy session from remote peer at address 127.10.11.254:35645
I20260927 00:34:03.870138 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.860968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.10.11.254:35645)
0927 00:34:03.861110 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:03.861117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:03.861414 (+ 297us) server_negotiation.cc:407] Connection header received
0927 00:34:03.861550 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:03.861553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:03.861612 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:03.861693 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:03.862648 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.863453 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.864800 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.865140 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.865240 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.866686 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:03.866690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:03.866693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:03.866717 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:03.868468 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.868931 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.868934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.868936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.869015 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:03.869288 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:03.869291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:03.869293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:03.869497 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:03.869590 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:03.869846 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:34:03.869951 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:34:03.870147 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:03.860844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57370)
0927 00:34:03.861294 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:03.861312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:03.861444 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:03.861737 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:03.861742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:03.861971 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:03.862494 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.862509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.863576 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.863583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.864666 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:03.864680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:03.865266 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:03.865282 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:03.865417 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:03.865984 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:03.866001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:03.866530 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:03.868581 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.868585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.868587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.868834 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.869107 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:03.869110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:03.869112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:03.869199 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:03.869589 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:03.869592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:03.869683 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:34:03.869784 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:34:03.869924 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":79,"threads_started":1}
I20260927 00:34:03.871027 16411 tablet_copy_service.cc:140] P c76f503a46ff4540985caceb448ce3e7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fa5126e3120f4e01bf7835377955f83e ({username='oryx', principal='oryx/127.10.11.253@KRBTEST.COM'} at 127.0.0.1:57370)
I20260927 00:34:03.871193 16411 tablet_copy_service.cc:161] P c76f503a46ff4540985caceb448ce3e7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fa5126e3120f4e01bf7835377955f83e at {username='oryx', principal='oryx/127.10.11.253@KRBTEST.COM'} at 127.0.0.1:57370: session id = fa5126e3120f4e01bf7835377955f83e-00000000000000000000000000000000
I20260927 00:34:03.872849 16411 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:34:03.874420 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:34:03.876492 16855 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: tablet copy: Starting download of 0 data blocks...
I20260927 00:34:03.876665 16855 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: tablet copy: Starting download of 1 WAL segments...
I20260927 00:34:03.878355 16855 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:34:03.880054 16411 tablet_copy_service.cc:342] P c76f503a46ff4540985caceb448ce3e7: Request end of tablet copy session fa5126e3120f4e01bf7835377955f83e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.11.253@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 00:34:03.880164 16411 tablet_copy_service.cc:433] P c76f503a46ff4540985caceb448ce3e7: ending tablet copy session fa5126e3120f4e01bf7835377955f83e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fa5126e3120f4e01bf7835377955f83e
I20260927 00:34:03.886885 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:03.888551 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:03.891132 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.891348 16855 server_base.cc:1034] running on GCE node
W20260927 00:34:03.894567 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:03.894841 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:03.895112 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:03.896248 16855 hybrid_clock.cc:648] HybridClock initialized: now 1790469243896231 us; error 36 us; skew 500 ppm
Sep 27 00:34:03 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469243, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:03.897816 16855 init.cc:377] Logged in from keytab as oryx/127.10.11.253@KRBTEST.COM (short username oryx)
I20260927 00:34:03.898325 16855 webserver.cc:460] Webserver started at http://127.10.11.253:45239/ using document root <none> and password file <none>
I20260927 00:34:03.898520 16855 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:03.898564 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:03.899925 16855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:03.900892 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:03.901110 16855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:03.901181 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "fa5126e3120f4e01bf7835377955f83e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "fe916697841c202dba988c24a711d934"
server_key_iv: "538bd1a3487db0dcf771bc1905572d8a"
server_key_version: "encryptionkey@0"
I20260927 00:34:03.901247 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:03.934407 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:03.936564 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:03.941437 16855 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.253:33761
I20260927 00:34:03.941538 17026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.253:33761 every 8 connection(s)
I20260927 00:34:03.942140 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
I20260927 00:34:03.947681 17027 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:34:03.948688 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Bootstrap starting.
I20260927 00:34:03.960108 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:34:03.960438 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: Bootstrap complete.
I20260927 00:34:03.961100 17027 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } }
I20260927 00:34:03.961267 17027 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa5126e3120f4e01bf7835377955f83e, State: Initialized, Role: LEARNER
I20260927 00:34:03.961404 17027 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } }
I20260927 00:34:03.961908 17027 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:03.965687 17041 catalog_manager.cc:1259] Loaded cluster ID: 72017ef12c144e48a44494858c9392df
I20260927 00:34:03.965745 17041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: loading cluster ID for follower catalog manager: success
I20260927 00:34:03.966200 17030 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:03.966315 17030 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: LEARNER
I20260927 00:34:03.967760 17041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: acquiring CA information for follower catalog manager: success
I20260927 00:34:03.969039 17041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:34:03.991533 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:03.991753 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:03.991791 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:03.995702 16957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:03.995744 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:03.995764 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:03.995779 16957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:03.995795 16957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:04.000910 16957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.252.keytab
--principal=oryx/127.10.11.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.11.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:04.001906 16957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:04.002920 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:04.010715 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.010680 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.010757 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.010933 16957 server_base.cc:1034] running on GCE node
I20260927 00:34:04.011590 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:04.012634 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:04.013797 16957 hybrid_clock.cc:648] HybridClock initialized: now 1790469244013762 us; error 43 us; skew 500 ppm
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:04.016350 16957 init.cc:377] Logged in from keytab as oryx/127.10.11.252@KRBTEST.COM (short username oryx)
I20260927 00:34:04.017097 16957 webserver.cc:460] Webserver started at http://127.10.11.252:33055/ using document root <none> and password file <none>
I20260927 00:34:04.017572 16957 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.017627 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.017773 16957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:04.019670 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/instance:
uuid: "9cfd3fe29685423d97bebf11a63e99c8"
format_stamp: "Formatted at 2026-09-27 00:34:04 on dist-test-slave-6gxc"
server_key: "e584a7bfd162259771fc70a9e796a55c"
server_key_iv: "cbc891283286d2bc741f0c285a0c9e35"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.020138 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal/instance:
uuid: "9cfd3fe29685423d97bebf11a63e99c8"
format_stamp: "Formatted at 2026-09-27 00:34:04 on dist-test-slave-6gxc"
server_key: "e584a7bfd162259771fc70a9e796a55c"
server_key_iv: "cbc891283286d2bc741f0c285a0c9e35"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.023546 16957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 00:34:04.025619 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.026893 16957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 00:34:04.027033 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "9cfd3fe29685423d97bebf11a63e99c8"
format_stamp: "Formatted at 2026-09-27 00:34:04 on dist-test-slave-6gxc"
server_key: "e584a7bfd162259771fc70a9e796a55c"
server_key_iv: "cbc891283286d2bc741f0c285a0c9e35"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.027137 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.045089 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:04.047814 16957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:04.048002 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:04.054519 16957 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.252:34887
I20260927 00:34:04.054773 17108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.252:34887 every 8 connection(s)
I20260927 00:34:04.055449 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
I20260927 00:34:04.058019 17109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:04.061450 17109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:34:04.063131 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 16957
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.252@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
I20260927 00:34:04.074410 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.063343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.10.11.254:35645)
0927 00:34:04.063493 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.063498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.064324 (+ 826us) server_negotiation.cc:407] Connection header received
0927 00:34:04.065348 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.065351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.065407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.065490 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.066318 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.066842 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.067824 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.068065 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.068176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.070514 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.070520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.070523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.070553 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.072176 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.072743 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.072745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.072747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.072830 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.073204 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.073207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.073209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.073443 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.073550 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.074083 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.074230 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:34:04.075317 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.063714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57376)
0927 00:34:04.064186 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.064235 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.065150 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.065541 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.065546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.065877 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.066182 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.066196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.066968 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.066971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.067693 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.067702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.068205 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.068208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.068369 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.068927 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.068950 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.070377 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.072280 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.072290 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.072299 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.072645 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.072919 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.072922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.072924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.073081 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.073565 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.073571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.073857 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.074022 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.074382 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260927 00:34:04.075317 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.064591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.10.11.252:34887)
0927 00:34:04.064839 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.064849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.064885 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:34:04.065165 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.065186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.065403 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.065586 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.066394 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.067230 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.068061 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.068411 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.068530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.071011 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.071016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.071027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.071056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.072710 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.073168 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.073176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.073181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.073266 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.073563 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.073566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.073568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.073839 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.074031 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.074268 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.074450 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260927 00:34:04.076964 17109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.252@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:04.084710 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.077366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44512)
0927 00:34:04.077464 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.077479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.077546 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.077952 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.077955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.078157 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.078245 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.078250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.079104 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.079110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.079619 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.079626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.080059 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.080062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.080152 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.080532 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.080542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.081669 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.083349 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.083355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.083357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.083539 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.083808 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.083811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.083812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.083890 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.084255 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.084258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.084340 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.084452 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.084563 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 00:34:04.084822 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.077493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.10.11.253:33761)
0927 00:34:04.077740 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.077744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.077755 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:34:04.077788 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.077790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.077825 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.077921 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.078353 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.079001 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.079729 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.079955 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.080063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.081808 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.081811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.081813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.081833 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.083250 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.083630 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.083636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.083638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.083712 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.083981 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.083986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.083987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.084145 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.084265 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.084501 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.084608 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260927 00:34:04.085706 17109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.252@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
I20260927 00:34:04.092730 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.086178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.0.0.1:35364)
0927 00:34:04.086318 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.086331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.086400 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.086607 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.086610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.086756 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.086807 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.086811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.087500 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.087506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.087979 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.087985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.088411 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.088414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.088529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.088856 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.088866 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.089873 (+ 1007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.091402 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.091408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.091411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.091593 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.091850 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.091852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.091854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.091928 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.092258 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.092261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.092335 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.092453 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.092567 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":27}
I20260927 00:34:04.092730 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.086260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.10.11.252:34887)
0927 00:34:04.086362 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.086367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.086379 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:04.086480 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.086484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.086515 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.086601 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.086913 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.087381 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.088081 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.088316 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.088414 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.090038 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.090041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.090043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.090074 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.091305 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.091680 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.091686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.091689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.091759 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.092011 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.092017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.092019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.092164 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.092257 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.092497 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.092615 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":39}
I20260927 00:34:04.095669 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Bootstrap starting.
I20260927 00:34:04.097878 17109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:04.098632 17109 log.cc:824] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:04.100389 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: No bootstrap required, opened a new log
I20260927 00:34:04.103410 17109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } }
I20260927 00:34:04.103593 17109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:04.103668 17109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfd3fe29685423d97bebf11a63e99c8, State: Initialized, Role: FOLLOWER
I20260927 00:34:04.104063 17109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } }
I20260927 00:34:04.104480 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } } }
I20260927 00:34:04.104584 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:04.105149 17109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:34:04.109395 17126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:34:04.109495 17126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:34:04.112643 17115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:04.112648 16957 master_runner.cc:428] Detected that this master 9cfd3fe29685423d97bebf11a63e99c8 is joining an existing cluster
I20260927 00:34:04.112758 16957 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.11.252:34887
I20260927 00:34:04.112819 17115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } }
I20260927 00:34:04.113703 17115 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645), fa5126e3120f4e01bf7835377955f83e (127.10.11.253:33761)
I20260927 00:34:04.114169 16401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cfd3fe29685423d97bebf11a63e99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76f503a46ff4540985caceb448ce3e7" is_pre_election: true
I20260927 00:34:04.114169 17001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cfd3fe29685423d97bebf11a63e99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5126e3120f4e01bf7835377955f83e" is_pre_election: true
I20260927 00:34:04.114331 17001 raft_consensus.cc:1799] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Handling vote request from an unknown peer 9cfd3fe29685423d97bebf11a63e99c8
I20260927 00:34:04.114459 17001 raft_consensus.cc:2396] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9cfd3fe29685423d97bebf11a63e99c8 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 00:34:04.114895 17058 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cfd3fe29685423d97bebf11a63e99c8; no voters: c76f503a46ff4540985caceb448ce3e7, fa5126e3120f4e01bf7835377955f83e
I20260927 00:34:04.115311 17115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:04.116240 17115 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:34:04.125945 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.117729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.10.11.254:35645)
0927 00:34:04.117844 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.117848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.117859 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:34:04.117889 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.117891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.117931 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.118019 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.118468 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.119161 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.120613 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.120952 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.121027 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.122420 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.122424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.122426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.122452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.124274 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.124742 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.124745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.124747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.124835 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.125138 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.125141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.125143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.125324 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.125391 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.125713 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.125815 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:34:04.125965 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.114867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57392)
0927 00:34:04.115190 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.115205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.115290 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.118036 (+ 2746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.118039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.118257 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.118349 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.118354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.119357 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.119360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.120471 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.120482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.121179 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.121183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.121291 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.121798 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.121818 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.122266 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.124386 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.124390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.124391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.124636 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.124955 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.124958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.124960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.125041 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.125458 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.125462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.125560 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.125661 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.125781 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260927 00:34:04.128501 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16453
I20260927 00:34:04.131111 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.122075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.10.11.253:33761)
0927 00:34:04.122212 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.122218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.122703 (+ 485us) server_negotiation.cc:407] Connection header received
0927 00:34:04.122861 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.122864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.122911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.123001 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.123708 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.124490 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.125614 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.125962 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.126057 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.127272 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.127276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.127278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.127302 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.128969 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.129733 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.129739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.129743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.129832 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.130140 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.130146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.130151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.130352 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.130456 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.130825 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.130956 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:34:04.131675 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.122292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44520)
0927 00:34:04.122620 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.122634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.122761 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.123241 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.123246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.123447 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.123575 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.123581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.124631 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.124634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.125499 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.125506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.126202 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.126205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.126302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.126746 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.126760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.127142 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.129421 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.129423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.129425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.129648 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.129929 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.129933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.129935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.130021 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.130503 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.130510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.130602 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.131448 (+ 846us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.131563 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260927 00:34:04.135079 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:38955
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:35645
--webserver_port=35175
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469242, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.254@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
I20260927 00:34:04.154955 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.119743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.0.0.1:35368)
0927 00:34:04.120256 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.120268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.120386 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.120663 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.120667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.120903 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.121009 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.121016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.122275 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.122278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.123656 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.123666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.124267 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.124270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.124361 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.150558 (+ 26197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.150576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.151097 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.153286 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.153296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.153571 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.153819 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.153824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.153894 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.154427 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.154429 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.154525 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.154621 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.154755 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"spinlock_wait_cycles":3456,"thread_start_us":72,"threads_started":1}
I20260927 00:34:04.155212 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.130596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.0.0.1:35382)
0927 00:34:04.130821 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.130835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.130896 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.135082 (+ 4186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.135085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.135269 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.135389 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.135399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.136397 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.136402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.137292 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.137305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.137869 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.137872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.137954 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.138448 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.138459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.139605 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.153631 (+ 14026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.153639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.153887 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.154233 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.154236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.154238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.154320 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.154729 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.154733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.154820 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.154905 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.155040 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 00:34:04.155460 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.120004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.10.11.252:34887)
0927 00:34:04.120090 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.120094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.120329 (+ 235us) server_negotiation.cc:407] Connection header received
0927 00:34:04.120476 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.120479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.120513 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.120556 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.121117 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.122160 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.123770 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.124091 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.124152 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.151238 (+ 27086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.151243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.151247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.151278 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.153170 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153660 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.153663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.153665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.153729 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153998 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.154000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.154002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.154304 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.154392 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.155224 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.155325 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":1792}
I20260927 00:34:04.155680 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.131078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.10.11.252:34887)
0927 00:34:04.134817 (+ 3739us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.134825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.134837 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:04.134905 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.134908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.134954 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.135042 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.135529 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.136300 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.137415 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.137767 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.137872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.139743 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.139746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.139748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.139766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.153514 (+ 13748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.153998 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.154000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.154002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.154130 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.154448 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.154450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.154452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.154628 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.155351 (+ 723us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.155416 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.155519 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3675,"spinlock_wait_cycles":2304,"thread_start_us":76,"threads_started":1}
I20260927 00:34:04.156863 16391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.11.252:34887: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8"
member_type: NON_VOTER
last_known_addr {
host: "127.10.11.252"
port: 34887
}
attrs {
promote: true
}
}
dest_uuid: "c76f503a46ff4540985caceb448ce3e7"
cas_config_opid_index: 5
I20260927 00:34:04.157371 16391 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
I20260927 00:34:04.158406 16951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.158983 17083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:34:04.159963 16951 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.11.252:34887
I20260927 00:34:04.160115 16951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:04.160733 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.160851 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.161511 16957 master.cc:537] Master@127.10.11.252:34887 shutting down...
I20260927 00:34:04.163796 17083 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
W20260927 00:34:04.163904 17083 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
I20260927 00:34:04.165264 17138 mvcc.cc:205] Tried to move back new op lower bound from 7333762018704936960 to 7333762018673057792. Current Snapshot: MvccSnapshot[applied={T|T < 7333762018704936960}]
I20260927 00:34:04.171205 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.161096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44524)
0927 00:34:04.161547 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.161561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.161648 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.166963 (+ 5315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.166966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.166986 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:04.167066 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.167072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.168908 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.168912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.169657 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.169665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.170784 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.170789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.170900 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.170952 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.171027 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.171097 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260927 00:34:04.171051 17083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa5126e3120f4e01bf7835377955f83e (127.10.11.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:04.171784 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.165575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.10.11.253:33761)
0927 00:34:04.166123 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.166127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.166141 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:04.166188 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.166191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.166249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.166342 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:04.168121 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.168751 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.169784 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.170448 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.170558 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.170721 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.171557 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.171675 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260927 00:34:04.172181 17001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:34:04.172626 17137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:34:04.174296 17001 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.176292 17137 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: attempting to promote NON_VOTER fa5126e3120f4e01bf7835377955f83e to VOTER
I20260927 00:34:04.176314 17030 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76f503a46ff4540985caceb448ce3e7. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.176409 17030 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: LEARNER
I20260927 00:34:04.176492 17030 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.176564 17030 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: LEARNER
I20260927 00:34:04.176702 17137 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
I20260927 00:34:04.177266 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76f503a46ff4540985caceb448ce3e7. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:04.177356 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 00:34:04.177438 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:04.177520 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 00:34:04.177583 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: NON_VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: true } } }
I20260927 00:34:04.177649 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 00:34:04.178257 17001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 LEARNER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:34:04.178563 16951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 00:34:04.179096 16376 consensus_peers.cc:487] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 -> Peer 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252:34887): Couldn't send request to peer 9cfd3fe29685423d97bebf11a63e99c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:34:04.180187 16951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fa5126e3120f4e01bf7835377955f83e (127.10.11.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.181099 16957 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 00:34:04.181336 16957 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 00:34:04.181427 16957 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: stopping tablet replica
I20260927 00:34:04.181591 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.181675 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.181802 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.181874 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.182487 17000 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fa5126e3120f4e01bf7835377955f83e (127.10.11.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.183748 17144 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.183857 17144 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:04.196628 16957 master.cc:552] Master@127.10.11.252:34887 shutdown complete.
I20260927 00:34:04.196687 16957 master_runner.cc:305] Clearing existing system tablet
I20260927 00:34:04.196898 16957 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.196970 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.198184 16957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:34:04.199074 17147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.199298 16957 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.199421 16957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:04.199461 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "9cfd3fe29685423d97bebf11a63e99c8"
format_stamp: "Formatted at 2026-09-27 00:34:04 on dist-test-slave-6gxc"
server_key: "e584a7bfd162259771fc70a9e796a55c"
server_key_iv: "cbc891283286d2bc741f0c285a0c9e35"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.200735 16957 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:04.202878 16957 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:34:04.202982 16957 log.cc:1196] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:34:04.203259 16957 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Deleting consensus metadata
I20260927 00:34:04.203415 16957 master_runner.cc:315] Copying system tablet from 127.10.11.254:35645
I20260927 00:34:04.204394 16957 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.11.254:35645
I20260927 00:34:04.212167 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.204951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.10.11.254:35645)
0927 00:34:04.205085 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.205089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.205300 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:34:04.205420 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.205423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.205472 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.205564 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.205993 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.206600 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.207462 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.207719 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.207838 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.209053 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.209056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.209059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.209082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.210614 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.211027 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.211032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.211034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.211117 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.211381 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.211386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.211388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.211560 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.211664 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.211905 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.212001 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:34:04.212179 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.204884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57400)
0927 00:34:04.205214 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.205225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.205343 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.205578 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.205581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.205789 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.205883 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.205888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.206705 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.206711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.207340 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.207348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.207838 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.207841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.207944 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.208458 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.208479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.208916 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.210732 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.210735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.210738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.210934 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.211211 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.211214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.211217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.211297 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.211654 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.211657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.211754 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.211857 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.211996 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260927 00:34:04.212973 16411 tablet_copy_service.cc:140] P c76f503a46ff4540985caceb448ce3e7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9cfd3fe29685423d97bebf11a63e99c8 ({username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:57400)
I20260927 00:34:04.213079 16411 tablet_copy_service.cc:161] P c76f503a46ff4540985caceb448ce3e7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9cfd3fe29685423d97bebf11a63e99c8 at {username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:57400: session id = 9cfd3fe29685423d97bebf11a63e99c8-00000000000000000000000000000000
I20260927 00:34:04.214015 16411 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:34:04.214994 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:34:04.216487 16957 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: tablet copy: Starting download of 0 data blocks...
I20260927 00:34:04.216655 16957 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: tablet copy: Starting download of 1 WAL segments...
I20260927 00:34:04.217850 16957 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:34:04.219256 16411 tablet_copy_service.cc:342] P c76f503a46ff4540985caceb448ce3e7: Request end of tablet copy session 9cfd3fe29685423d97bebf11a63e99c8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:57400
I20260927 00:34:04.219329 16411 tablet_copy_service.cc:433] P c76f503a46ff4540985caceb448ce3e7: ending tablet copy session 9cfd3fe29685423d97bebf11a63e99c8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9cfd3fe29685423d97bebf11a63e99c8
I20260927 00:34:04.225301 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:04.226951 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.227165 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.227315 16957 server_base.cc:1034] running on GCE node
W20260927 00:34:04.230513 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.230706 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:04.230964 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:04.232085 16957 hybrid_clock.cc:648] HybridClock initialized: now 1790469244232065 us; error 35 us; skew 500 ppm
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:04.233462 16957 init.cc:377] Logged in from keytab as oryx/127.10.11.252@KRBTEST.COM (short username oryx)
I20260927 00:34:04.233891 16957 webserver.cc:460] Webserver started at http://127.10.11.252:41781/ using document root <none> and password file <none>
I20260927 00:34:04.234090 16957 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.234156 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.235180 16957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:34:04.236027 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.236239 16957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:34:04.236326 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "9cfd3fe29685423d97bebf11a63e99c8"
format_stamp: "Formatted at 2026-09-27 00:34:04 on dist-test-slave-6gxc"
server_key: "e584a7bfd162259771fc70a9e796a55c"
server_key_iv: "cbc891283286d2bc741f0c285a0c9e35"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.236400 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.257141 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:04.259013 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:04.263819 16957 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.252:34887
I20260927 00:34:04.263940 17216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.252:34887 every 8 connection(s)
I20260927 00:34:04.264336 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
I20260927 00:34:04.265729 17217 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:34:04.266633 17217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Bootstrap starting.
I20260927 00:34:04.272480 17217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:34:04.272738 17217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Bootstrap complete.
I20260927 00:34:04.273247 17217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
I20260927 00:34:04.273365 17217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cfd3fe29685423d97bebf11a63e99c8, State: Initialized, Role: LEARNER
I20260927 00:34:04.273474 17217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } }
I20260927 00:34:04.273761 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.273916 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:34:04.273916 17217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:04.277113 17231 catalog_manager.cc:1259] Loaded cluster ID: 72017ef12c144e48a44494858c9392df
I20260927 00:34:04.277163 17231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: loading cluster ID for follower catalog manager: success
I20260927 00:34:04.278702 17231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: acquiring CA information for follower catalog manager: success
I20260927 00:34:04.279747 17231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:34:04.287947 17136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:04.288235 17136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:04.288295 17136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:04.293973 17136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:04.294037 17136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:04.294138 17136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:04.301513 17136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=35175
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:04.302670 17136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:04.303937 17136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:04.314733 17239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.314702 17237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.314927 17236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.315162 17136 server_base.cc:1034] running on GCE node
I20260927 00:34:04.315546 17136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:04.316165 17136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:04.317345 17136 hybrid_clock.cc:648] HybridClock initialized: now 1790469244317327 us; error 39 us; skew 500 ppm
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:04.319926 17136 init.cc:377] Logged in from keytab as oryx/127.10.11.193@KRBTEST.COM (short username oryx)
I20260927 00:34:04.320676 17136 webserver.cc:460] Webserver started at http://127.10.11.193:35175/ using document root <none> and password file <none>
I20260927 00:34:04.321161 17136 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.321226 17136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.325006 17136 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:04.327042 17246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.328362 17136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:04.328454 17136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "b83e20ead07c49c18bb6b753d04d350e"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "06d2f7bb5fa7631cf4c1bfedccd4cd40"
server_key_iv: "d42e8ce41ad2c863a57c0b2bb78e2783"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.328991 17136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.345716 17136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:04.348456 17136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:04.348656 17136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:04.349225 17136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:04.350067 17136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:04.350136 17136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.350180 17136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:04.350224 17136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.353371 16716 heartbeater.cc:503] Master 127.10.11.254:35645 was elected leader, sending a full tablet report...
I20260927 00:34:04.359575 17136 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:38955
I20260927 00:34:04.359608 17359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:38955 every 8 connection(s)
I20260927 00:34:04.360525 17136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:04.364545 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 17136
I20260927 00:34:04.364749 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16587
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:04.375707 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:35729
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:35645
--webserver_port=35457
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:34:04.379400 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.364275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:35135 (local address 127.10.11.254:35645)
0927 00:34:04.364396 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.364399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.364411 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:04.365065 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.365071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.365131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.365190 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.367216 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368117 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370239 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.370628 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370713 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.375163 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.375169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.375174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.375203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.376980 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.377518 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.377521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.377523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.377604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.378006 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.378009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.378010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.378251 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.378345 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.379050 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.379176 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:34:04.380463 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.362503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:48397 (local address 127.10.11.252:34887)
0927 00:34:04.362943 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.362955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.364114 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 00:34:04.366160 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.366164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.366231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.366319 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.367218 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368399 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370256 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.370629 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370725 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.374970 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.374974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.374977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.375002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.376828 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.377518 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.377521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.377523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.377604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.378004 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.378007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.378009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.378295 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.378378 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.380140 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.380283 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":208,"threads_started":1}
I20260927 00:34:04.380720 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.363930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:35135)
0927 00:34:04.364183 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.364200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.364835 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.365613 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.365625 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.366268 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.367046 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.367062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368290 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.368293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.370044 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.370055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.370886 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.371225 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.372109 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.372152 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.375015 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.377085 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.377089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.377091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.377405 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.377705 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.377708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.377710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.377896 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.378415 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.378421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.378756 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.378951 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.379339 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":452,"spinlock_wait_cycles":17024,"thread_start_us":82,"threads_started":1}
I20260927 00:34:04.380950 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.363053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:59679 (local address 127.10.11.253:33761)
0927 00:34:04.363244 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.363251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.363881 (+ 630us) server_negotiation.cc:407] Connection header received
0927 00:34:04.365593 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.365600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.365668 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.365732 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.367236 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368157 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370282 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.370711 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370814 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.376740 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.376748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.376754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.376792 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.378892 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.379478 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.379482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.379484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.379576 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.379927 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.379931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.379933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.380163 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.380263 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.380629 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.380744 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:34:04.381592 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.362845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.10.11.193:59679)
0927 00:34:04.363764 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.363780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.364822 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.365902 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.365908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.366270 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.367038 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.367062 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368333 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.368339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.370115 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.370131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.371015 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.371022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.371218 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.372354 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.372376 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.376549 (+ 4173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.379049 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.379053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.379056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.379347 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.379714 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.379718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.379721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.379813 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.380326 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.380329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.380418 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.381196 (+ 778us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.381372 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":387,"spinlock_wait_cycles":8320,"thread_start_us":121,"threads_started":1}
I20260927 00:34:04.382136 17360 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
I20260927 00:34:04.382316 17362 heartbeater.cc:347] Connected to a master server at 127.10.11.253:33761
I20260927 00:34:04.382404 17360 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.382404 17362 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.382167 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.362845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.10.11.193:48397)
0927 00:34:04.363602 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.363652 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.365976 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.366456 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.366459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.366632 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.367048 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.367061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.368525 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.368529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.370120 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.370131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.370886 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.370890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.371201 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.372178 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.372193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.374671 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.376978 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.376984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.376997 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.377405 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.377705 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.377708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.377710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.377896 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.378415 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.378421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.379923 (+ 1502us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.381274 (+ 1351us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.381950 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":2110,"spinlock_wait_cycles":51456,"thread_start_us":671,"threads_started":1}
I20260927 00:34:04.383035 17360 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:04.383033 17362 heartbeater.cc:511] Master 127.10.11.253:33761 requested a full tablet report, sending...
I20260927 00:34:04.383528 17361 heartbeater.cc:347] Connected to a master server at 127.10.11.252:34887
I20260927 00:34:04.383595 17361 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.383816 17361 heartbeater.cc:511] Master 127.10.11.252:34887 requested a full tablet report, sending...
I20260927 00:34:04.384469 16391 ts_manager.cc:194] Re-registered known tserver with Master: b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
I20260927 00:34:04.384733 16991 ts_manager.cc:194] Registered new tserver with Master: b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
I20260927 00:34:04.384821 17181 ts_manager.cc:194] Registered new tserver with Master: b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
I20260927 00:34:04.385264 16391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:35135
W20260927 00:34:04.486889 17368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:04.487093 17368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:04.487131 17368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:04.490985 17368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:04.491029 17368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:04.491082 17368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:04.496001 17368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:35729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=35457
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:04.496814 17368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:04.497637 17368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:04.506832 17376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.507005 17368 server_base.cc:1034] running on GCE node
W20260927 00:34:04.506860 17374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.506844 17373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.507517 17368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:04.508100 17368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:04.509299 17368 hybrid_clock.cc:648] HybridClock initialized: now 1790469244509264 us; error 52 us; skew 500 ppm
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:04.512044 17368 init.cc:377] Logged in from keytab as oryx/127.10.11.194@KRBTEST.COM (short username oryx)
I20260927 00:34:04.512825 17368 webserver.cc:460] Webserver started at http://127.10.11.194:35457/ using document root <none> and password file <none>
I20260927 00:34:04.513341 17368 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.513439 17368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.516971 17368 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:04.518875 17383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.519984 17368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:04.520118 17368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "d4bf4552f23f4064a5ecbab3836a5fe3"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "0b897b9c8ac07112106da16a32577665"
server_key_iv: "73786dc5e4929d8eec932fe4683006e9"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.520637 17368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.539484 17368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:04.542837 17368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:04.543010 17368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:04.543613 17368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:04.544528 17368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:04.544584 17368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.544629 17368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:04.544657 17368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.553844 17368 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:35729
I20260927 00:34:04.553871 17496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:35729 every 8 connection(s)
I20260927 00:34:04.554844 17368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:04.560619 16850 heartbeater.cc:503] Master 127.10.11.254:35645 was elected leader, sending a full tablet report...
I20260927 00:34:04.563066 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 17368
I20260927 00:34:04.563251 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16721
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:04.570292 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.558090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54343 (local address 127.10.11.253:33761)
0927 00:34:04.558231 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.558235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.558250 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:34:04.558696 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.558700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.558752 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.558806 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.560015 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560831 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562516 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.562830 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562895 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.566529 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.566538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.566542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.566572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.568112 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568652 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.568658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.568663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.568745 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.569075 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.569081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.569086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.569346 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.569441 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.569903 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.570037 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:34:04.570451 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.556520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:37629 (local address 127.10.11.254:35645)
0927 00:34:04.556640 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.556645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.557617 (+ 972us) server_negotiation.cc:407] Connection header received
0927 00:34:04.558671 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.558674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.558730 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.558839 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.559994 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560546 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562305 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.562642 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562748 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.566042 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.566048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.566054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.566087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.567958 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568535 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.568539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.568541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.568631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.569002 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.569007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.569011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.569201 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.569320 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.570208 (+ 888us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.570309 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:34:04.571102 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:35921
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:35645
--webserver_port=44107
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:34:04.571142 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.556854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.10.11.194:54343)
0927 00:34:04.557724 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.557735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.558499 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.559006 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.559009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.559399 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.559858 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.559873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560956 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.560959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.562404 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.562413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.563040 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.563043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.563200 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.564920 (+ 1720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.564940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.566372 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.568224 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.568235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.568541 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.568840 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.568846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.568935 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.569436 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.569439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.569599 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.569808 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.570189 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":261,"spinlock_wait_cycles":22784,"thread_start_us":126,"threads_started":1}
I20260927 00:34:04.571574 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.556854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:37629)
0927 00:34:04.557425 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.557488 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.558477 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.559000 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.559006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.559382 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.559855 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.559867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560707 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.560711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.562190 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.562198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.562785 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.563004 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.564132 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.564157 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.565858 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.568083 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.568100 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.568433 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.568708 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.568714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.568719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.568891 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.569315 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.569321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.569599 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.569808 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.571044 (+ 1236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1023,"spinlock_wait_cycles":5376,"thread_start_us":640,"threads_started":1}
I20260927 00:34:04.572690 17497 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
I20260927 00:34:04.572909 17497 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.573421 17499 heartbeater.cc:347] Connected to a master server at 127.10.11.253:33761
I20260927 00:34:04.573450 17497 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:04.573495 17499 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.573626 17499 heartbeater.cc:511] Master 127.10.11.253:33761 requested a full tablet report, sending...
I20260927 00:34:04.574611 16391 ts_manager.cc:194] Re-registered known tserver with Master: d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729)
I20260927 00:34:04.574868 16991 ts_manager.cc:194] Registered new tserver with Master: d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729)
I20260927 00:34:04.575462 16391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:37629
I20260927 00:34:04.578544 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.557838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.10.11.194:44475)
0927 00:34:04.558141 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.558154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.558499 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.559002 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.559007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.559426 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.559858 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.559873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560956 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.560959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.562260 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.562270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.563040 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.563043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.563160 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.571023 (+ 7863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.571043 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.573147 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.576776 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.576779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.576781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.577014 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.577389 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.577391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.577393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.577515 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.578030 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.578032 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.578196 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.578293 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.578417 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":171,"spinlock_wait_cycles":38656,"thread_start_us":87,"threads_started":1}
I20260927 00:34:04.578789 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.557982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:44475 (local address 127.10.11.252:34887)
0927 00:34:04.558107 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.558145 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.558207 (+ 62us) server_negotiation.cc:407] Connection header received
0927 00:34:04.558696 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.558700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.558754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.558806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.560015 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.560798 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562407 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.562783 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.562875 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.573277 (+ 10402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.573284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.573290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.573322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.576648 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.577142 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.577149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.577154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.577252 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.577645 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.577652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.577656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.577913 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.578029 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.578412 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.578566 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:34:04.579332 17498 heartbeater.cc:347] Connected to a master server at 127.10.11.252:34887
I20260927 00:34:04.579427 17498 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.579644 17498 heartbeater.cc:511] Master 127.10.11.252:34887 requested a full tablet report, sending...
I20260927 00:34:04.580428 17181 ts_manager.cc:194] Registered new tserver with Master: d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729)
W20260927 00:34:04.680634 17504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:04.680866 17504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:04.680907 17504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:04.684968 17504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:04.685019 17504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:04.685103 17504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:04.690346 17504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:35921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=44107
--tserver_master_addrs=127.10.11.254:35645,127.10.11.253:33761,127.10.11.252:34887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:04.691404 17504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:04.692341 17504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:04.693542 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.689964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.10.11.252:34887)
0927 00:34:04.690087 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.690092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.690350 (+ 258us) server_negotiation.cc:407] Connection header received
0927 00:34:04.690518 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.690523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.690603 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.690699 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:04.690981 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.691532 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.692573 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.692993 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.693133 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.693278 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.693360 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.693423 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:34:04.693545 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.689849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.0.0.1:35398)
0927 00:34:04.690236 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.690251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.690386 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.690738 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.690741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.690763 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:04.690872 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.690878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.691642 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.691646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.692455 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.692468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.693139 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.693143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.693230 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.693251 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.693335 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.693418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":62,"threads_started":1}
I20260927 00:34:04.694018 17191 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 00:34:04.694319 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76f503a46ff4540985caceb448ce3e7. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.694461 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: LEARNER
W20260927 00:34:04.701740 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.701742 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:04.701750 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:04.701797 17504 server_base.cc:1034] running on GCE node
I20260927 00:34:04.702471 17504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:04.703039 17504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:04.704247 17504 hybrid_clock.cc:648] HybridClock initialized: now 1790469244704208 us; error 50 us; skew 500 ppm
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:04.706866 17504 init.cc:377] Logged in from keytab as oryx/127.10.11.195@KRBTEST.COM (short username oryx)
I20260927 00:34:04.707674 17504 webserver.cc:460] Webserver started at http://127.10.11.195:44107/ using document root <none> and password file <none>
I20260927 00:34:04.708182 17504 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:04.708273 17504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:04.711797 17504 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:04.713869 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.715070 17504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:04.715178 17504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "d294934e21a446bd8c8a872ca3266d0c"
format_stamp: "Formatted at 2026-09-27 00:34:03 on dist-test-slave-6gxc"
server_key: "3a2c7c8e7abec7b324fa389cb54b2c7d"
server_key_iv: "d6799b85c96c41a99adf373723f1d7b6"
server_key_version: "encryptionkey@0"
I20260927 00:34:04.715664 17504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:04.743551 17504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:04.746901 17504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:04.747087 17504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:04.747681 17504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:04.748564 17504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:04.748610 17504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.748674 17504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:04.748704 17504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:04.757282 17504 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:35921
I20260927 00:34:04.757355 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:35921 every 8 connection(s)
I20260927 00:34:04.758335 17504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:04.758832 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 17504
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.252@KRBTEST.COM
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:04.775437 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.761423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.10.11.253:33761)
0927 00:34:04.762930 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.762934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.762946 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:04.762985 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.762987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.763023 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.763183 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.764307 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.765169 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.766406 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.766754 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.766867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.770698 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.770706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.770711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.770742 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.772946 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.773443 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.773447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.773448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.773526 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.773902 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.773905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.773907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.774191 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.775018 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.775102 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.775229 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":74,"threads_started":1}
I20260927 00:34:04.776052 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.762323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:36757 (local address 127.10.11.252:34887)
0927 00:34:04.762627 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.762632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.762865 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:34:04.763764 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.763767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.763842 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.763922 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.765473 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.766251 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768102 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768438 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768518 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.771441 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.771444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.771447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.771471 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.773208 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.773874 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.773879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.773884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.773967 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.774396 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.774399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.774401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.774610 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.774707 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.775742 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.775868 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260927 00:34:04.777271 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.762410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.10.11.252:34887)
0927 00:34:04.762671 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.762684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.764144 (+ 1460us) server_negotiation.cc:407] Connection header received
0927 00:34:04.764177 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.764179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.764220 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.764299 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.765174 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.765972 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.767912 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768238 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768346 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.770197 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.770202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.770207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.770234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.774027 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.775282 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.775285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.775287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.775415 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.776192 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.776196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.776197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.776402 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.776515 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.777013 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.777120 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260927 00:34:04.778008 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.761891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.10.11.254:35645)
0927 00:34:04.763004 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.763009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.763555 (+ 546us) server_negotiation.cc:407] Connection header received
0927 00:34:04.764386 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.764389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.764431 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.764502 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.767339 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768619 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.770284 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.770653 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.770730 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.773275 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.773279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.773282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.773306 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.775889 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.776362 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.776366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.776368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.776456 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.776763 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.776770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.776787 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.777003 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.777116 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.777678 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.777812 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1052,"thread_start_us":75,"threads_started":1}
I20260927 00:34:04.778350 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.761922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.10.11.195:36757)
0927 00:34:04.762771 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.762784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.763271 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.764099 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.764108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.765091 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.765322 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.765344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.766425 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.766428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.767975 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.767985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768667 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.768917 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.769449 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.769468 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.771265 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.773332 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.773337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.773352 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.773747 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.774084 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.774086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.774088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.774269 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.774913 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.774924 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.775221 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.776383 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.776706 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":143,"thread_start_us":67,"threads_started":1}
I20260927 00:34:04.779894 17636 heartbeater.cc:347] Connected to a master server at 127.10.11.252:34887
I20260927 00:34:04.780146 17636 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.780161 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.760846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:60061 (local address 127.10.11.254:35645)
0927 00:34:04.760990 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.760996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.762356 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 00:34:04.763488 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.763491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.763546 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.763633 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.765480 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.766237 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768088 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768437 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768534 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.774636 (+ 6102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.774641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.774644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.774671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.776917 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.777837 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.777840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.777842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.777912 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.778356 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.778362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.778366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.778548 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.778613 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.779725 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.779839 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":54}
I20260927 00:34:04.780658 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.760647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:60061)
0927 00:34:04.762254 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.762273 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.763268 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.764101 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.764111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.764647 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.765325 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.765343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.766420 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.766424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.767975 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.767983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.768874 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.768884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.769023 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.769881 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.769902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.774479 (+ 4577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.777253 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.777256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.777258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.777725 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.778138 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.778141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.778143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.778232 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.778682 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.778685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.778770 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.780327 (+ 1557us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.780473 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"mutex_wait_us":369,"thread_start_us":104,"threads_started":1}
I20260927 00:34:04.780725 17636 heartbeater.cc:511] Master 127.10.11.252:34887 requested a full tablet report, sending...
I20260927 00:34:04.781139 17635 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35645
Sep 27 00:34:04 dist-test-slave-6gxc krb5kdc[16346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469244, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.253@KRBTEST.COM
I20260927 00:34:04.781212 17635 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.781368 17635 heartbeater.cc:511] Master 127.10.11.254:35645 requested a full tablet report, sending...
I20260927 00:34:04.781793 17181 ts_manager.cc:194] Registered new tserver with Master: d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:04.781872 16390 ts_manager.cc:194] Re-registered known tserver with Master: d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:04.782816 16390 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:60061
I20260927 00:34:04.781195 16391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:04.785126 16391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:04.785735 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.760636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:33267 (local address 127.10.11.253:33761)
0927 00:34:04.760760 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.760765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.762252 (+ 1487us) server_negotiation.cc:407] Connection header received
0927 00:34:04.763658 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.763661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.763707 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.763762 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:04.771387 (+ 7625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.772170 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.778406 (+ 6236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.778743 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.778829 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.781661 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:04.781665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:04.781667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:04.781690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:04.784110 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.784613 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.784616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.784618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.784701 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:04.784977 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:04.784980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:04.784981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:04.785143 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:04.785245 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.785504 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.785603 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:34:04.785748 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.760646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.10.11.195:33267)
0927 00:34:04.762088 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.762162 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.763548 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.764099 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.764108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.769264 (+ 5156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:04.771216 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.771227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.772814 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.772819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.778289 (+ 5470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.778299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.779001 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.779004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.779099 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.779544 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:04.779557 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:04.781521 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:04.784224 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.784230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.784232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.784506 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.784792 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:04.784796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:04.784797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:04.784881 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:04.785248 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:04.785253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:04.785339 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.785444 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.785583 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":658,"threads_started":1}
I20260927 00:34:04.786336 17637 heartbeater.cc:347] Connected to a master server at 127.10.11.253:33761
I20260927 00:34:04.786422 17637 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:04.786602 17637 heartbeater.cc:511] Master 127.10.11.253:33761 requested a full tablet report, sending...
I20260927 00:34:04.787297 16991 ts_manager.cc:194] Registered new tserver with Master: d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:04.787840 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: NON_VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: true } } }
I20260927 00:34:04.787935 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.789533 17653 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: attempting to promote NON_VOTER 9cfd3fe29685423d97bebf11a63e99c8 to VOTER
I20260927 00:34:04.789853 17653 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } }
I20260927 00:34:04.790916 17191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 LEARNER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:34:04.791126 17000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Refusing update from remote peer c76f503a46ff4540985caceb448ce3e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:34:04.791272 17653 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 00:34:04.791452 17508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 00:34:04.793079 17508 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.793442 17000 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.794579 17658 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.794602 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.794703 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.794690 17658 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.794807 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.794911 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:04.795056 17662 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.795161 17662 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:04.795887 17191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.797098 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c76f503a46ff4540985caceb448ce3e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:04.797215 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:04.806080 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.801030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35921 (local address 127.0.0.1:35600)
0927 00:34:04.801413 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.801426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.801523 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.802077 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.802079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.802094 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:04.802221 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.802228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.803983 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.803990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.804881 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.804890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.805682 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.805686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.805774 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.805797 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.805885 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.805947 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":78,"thread_start_us":87,"threads_started":1}
I20260927 00:34:04.806147 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.801132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.10.11.195:35921)
0927 00:34:04.801373 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.801386 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.801513 (+ 127us) server_negotiation.cc:407] Connection header received
0927 00:34:04.801699 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.801705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.801903 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.802085 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:04.802344 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.803856 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.805006 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.805571 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.805686 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.805817 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.805911 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.805992 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260927 00:34:04.807204 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.801038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:38955 (local address 127.0.0.1:41176)
0927 00:34:04.801874 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.801885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.801960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.802606 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.802611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.802628 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:04.802707 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.802713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.804014 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.804019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.805467 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.805477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.806776 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.806782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.806872 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.806900 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.806980 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.807060 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":140,"thread_start_us":95,"threads_started":1}
I20260927 00:34:04.807353 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.800430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:35729 (local address 127.0.0.1:34232)
0927 00:34:04.800730 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:04.800744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:04.800827 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:04.802726 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:04.802729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:04.802746 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:04.802838 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.802846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.804007 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.804010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.805467 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:04.805477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.806983 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.806987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:04.807086 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.807105 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:34:04.807172 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:34:04.807245 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":216}
I20260927 00:34:04.807585 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.801481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.10.11.193:38955)
0927 00:34:04.801778 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.801789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.801965 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:34:04.802045 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.802051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.802260 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.802449 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:04.802836 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.803925 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.805609 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.806487 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.806585 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.806716 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.807351 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.807450 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260927 00:34:04.807880 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:04.801024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.10.11.194:35729)
0927 00:34:04.802235 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:04.802247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:04.802271 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:34:04.802357 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:04.802365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:04.802591 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:04.802748 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:04.802932 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.803865 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.805602 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:04.806878 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:04.806988 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:04.807110 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:04.807193 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 00:34:04.807736 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":109,"threads_started":1}
I20260927 00:34:04.808869 17569 tablet_service.cc:1467] Processing CreateTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 (DEFAULT_TABLE table=test-table [id=a97e8fd0112d449484f0ecb60891b806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:04.809718 17569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9441cdf1c634e40b16570e4ee9eebd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:04.816349 17294 tablet_service.cc:1467] Processing CreateTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 (DEFAULT_TABLE table=test-table [id=a97e8fd0112d449484f0ecb60891b806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:04.816395 17431 tablet_service.cc:1467] Processing CreateTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 (DEFAULT_TABLE table=test-table [id=a97e8fd0112d449484f0ecb60891b806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:04.817107 17294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9441cdf1c634e40b16570e4ee9eebd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:04.817237 17431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9441cdf1c634e40b16570e4ee9eebd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:04.824115 17669 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Bootstrap starting.
I20260927 00:34:04.824865 17670 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Bootstrap starting.
I20260927 00:34:04.826390 17669 tablet_bootstrap.cc:654] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:04.826982 17670 tablet_bootstrap.cc:654] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:04.827140 17669 log.cc:824] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:04.827890 17670 log.cc:824] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:04.829015 17669 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: No bootstrap required, opened a new log
I20260927 00:34:04.829216 17669 ts_tablet_manager.cc:1397] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:34:04.829968 17670 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: No bootstrap required, opened a new log
I20260927 00:34:04.830189 17670 ts_tablet_manager.cc:1397] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:34:04.834064 17669 raft_consensus.cc:348] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.834396 17669 raft_consensus.cc:374] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:04.834478 17669 raft_consensus.cc:729] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bf4552f23f4064a5ecbab3836a5fe3, State: Initialized, Role: FOLLOWER
I20260927 00:34:04.835011 17669 consensus_queue.cc:260] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.835542 17670 raft_consensus.cc:348] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.835788 17670 raft_consensus.cc:374] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:04.835850 17670 raft_consensus.cc:729] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83e20ead07c49c18bb6b753d04d350e, State: Initialized, Role: FOLLOWER
I20260927 00:34:04.836160 17669 ts_tablet_manager.cc:1428] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:34:04.836308 17670 consensus_queue.cc:260] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.836568 17497 heartbeater.cc:503] Master 127.10.11.254:35645 was elected leader, sending a full tablet report...
I20260927 00:34:04.837138 17674 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Bootstrap starting.
I20260927 00:34:04.837277 17670 ts_tablet_manager.cc:1428] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:34:04.837584 17360 heartbeater.cc:503] Master 127.10.11.254:35645 was elected leader, sending a full tablet report...
I20260927 00:34:04.839504 17674 tablet_bootstrap.cc:654] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:04.840890 17674 log.cc:824] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:04.845690 17674 tablet_bootstrap.cc:492] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: No bootstrap required, opened a new log
I20260927 00:34:04.845896 17674 ts_tablet_manager.cc:1397] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 00:34:04.849072 17674 raft_consensus.cc:348] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.849264 17674 raft_consensus.cc:374] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:04.849308 17674 raft_consensus.cc:729] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d294934e21a446bd8c8a872ca3266d0c, State: Initialized, Role: FOLLOWER
I20260927 00:34:04.849761 17674 consensus_queue.cc:260] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:04.850507 17635 heartbeater.cc:503] Master 127.10.11.254:35645 was elected leader, sending a full tablet report...
I20260927 00:34:04.850690 17674 ts_tablet_manager.cc:1428] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 00:34:04.863812 17363 tablet.cc:2369] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:34:05.010767 17638 tablet.cc:2369] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:05.025808 17675 raft_consensus.cc:482] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:05.026021 17675 raft_consensus.cc:504] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:05.027195 17675 leader_election.cc:288] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729), d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:05.031100 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.027709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:60395 (local address 127.10.11.194:35729)
0927 00:34:05.027824 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.027829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.027987 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:34:05.028153 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.028155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.028194 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.028276 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:05.028552 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.029043 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030152 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.030545 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030677 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.030730 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.030898 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.030962 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:34:05.031114 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.027591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:40381 (local address 127.10.11.195:35921)
0927 00:34:05.027735 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.027740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.027942 (+ 202us) server_negotiation.cc:407] Connection header received
0927 00:34:05.028110 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.028112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.028154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.028239 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:05.028511 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.029016 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030167 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.030567 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.030743 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.030924 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.030990 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:34:05.031127 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.027484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35921 (local address 127.10.11.193:40381)
0927 00:34:05.027838 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:05.027858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:05.027990 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:05.028249 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:05.028252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:05.028266 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:05.028400 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:05.028406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.029125 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.029132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:05.030022 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:05.030031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.030695 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:05.030793 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.030818 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:34:05.030909 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:34:05.030980 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":74,"thread_start_us":83,"threads_started":1}
I20260927 00:34:05.031145 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.027484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:35729 (local address 127.10.11.193:60395)
0927 00:34:05.027877 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:05.027890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:05.028032 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:05.028288 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:05.028291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:05.028306 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:05.028447 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:05.028452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.029156 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.029160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:05.030010 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:05.030019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.030672 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.030675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:05.030767 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.030789 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:34:05.030885 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:34:05.030960 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":190,"thread_start_us":62,"threads_started":1}
I20260927 00:34:05.031730 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9441cdf1c634e40b16570e4ee9eebd5" candidate_uuid: "b83e20ead07c49c18bb6b753d04d350e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d294934e21a446bd8c8a872ca3266d0c" is_pre_election: true
I20260927 00:34:05.031823 17451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9441cdf1c634e40b16570e4ee9eebd5" candidate_uuid: "b83e20ead07c49c18bb6b753d04d350e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" is_pre_election: true
I20260927 00:34:05.032047 17451 raft_consensus.cc:2454] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83e20ead07c49c18bb6b753d04d350e in term 0.
I20260927 00:34:05.032047 17589 raft_consensus.cc:2454] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83e20ead07c49c18bb6b753d04d350e in term 0.
I20260927 00:34:05.032533 17249 leader_election.cc:302] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b83e20ead07c49c18bb6b753d04d350e, d294934e21a446bd8c8a872ca3266d0c, d4bf4552f23f4064a5ecbab3836a5fe3; no voters:
I20260927 00:34:05.032805 17675 raft_consensus.cc:2781] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:05.032900 17675 raft_consensus.cc:482] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:05.032954 17675 raft_consensus.cc:3037] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:05.033931 17675 raft_consensus.cc:504] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:05.034368 17675 leader_election.cc:288] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [CANDIDATE]: Term 1 election: Requested vote from peers d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729), d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921)
I20260927 00:34:05.034677 17451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9441cdf1c634e40b16570e4ee9eebd5" candidate_uuid: "b83e20ead07c49c18bb6b753d04d350e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3"
I20260927 00:34:05.034687 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9441cdf1c634e40b16570e4ee9eebd5" candidate_uuid: "b83e20ead07c49c18bb6b753d04d350e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d294934e21a446bd8c8a872ca3266d0c"
I20260927 00:34:05.034809 17451 raft_consensus.cc:3037] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:05.034806 17589 raft_consensus.cc:3037] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:05.035948 17451 raft_consensus.cc:2454] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83e20ead07c49c18bb6b753d04d350e in term 1.
I20260927 00:34:05.036000 17589 raft_consensus.cc:2454] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83e20ead07c49c18bb6b753d04d350e in term 1.
I20260927 00:34:05.036227 17247 leader_election.cc:302] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b83e20ead07c49c18bb6b753d04d350e, d4bf4552f23f4064a5ecbab3836a5fe3; no voters:
I20260927 00:34:05.036438 17675 raft_consensus.cc:2781] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:05.036650 17675 raft_consensus.cc:686] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 1 LEADER]: Becoming Leader. State: Replica: b83e20ead07c49c18bb6b753d04d350e, State: Running, Role: LEADER
I20260927 00:34:05.036983 17675 consensus_queue.cc:237] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } }
I20260927 00:34:05.040907 16390 catalog_manager.cc:5495] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e reported cstate change: term changed from 0 to 1, leader changed from <none> to b83e20ead07c49c18bb6b753d04d350e (127.10.11.193). New cstate: current_term: 1 leader_uuid: "b83e20ead07c49c18bb6b753d04d350e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } health_report { overall_health: UNKNOWN } } }
W20260927 00:34:05.057535 17500 tablet.cc:2369] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:05.099825 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.096288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.10.11.193:38955)
0927 00:34:05.096419 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.096423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.096436 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:05.096505 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.096507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.096555 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.096645 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:05.096931 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.097412 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.098319 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.098540 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.098665 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.098958 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:05.099481 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:05.099579 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.099670 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.099719 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:34:05.105988 17451 raft_consensus.cc:1261] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Refusing update from remote peer b83e20ead07c49c18bb6b753d04d350e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:05.105988 17589 raft_consensus.cc:1261] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Refusing update from remote peer b83e20ead07c49c18bb6b753d04d350e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:05.106881 17675 consensus_queue.cc:1035] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:05.107107 17680 consensus_queue.cc:1035] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:05.115130 17683 mvcc.cc:205] Tried to move back new op lower bound from 7333762027946283008 to 7333762027674202112. Current Snapshot: MvccSnapshot[applied={T|T < 7333762027946283008}]
I20260927 00:34:05.117787 17684 mvcc.cc:205] Tried to move back new op lower bound from 7333762027946283008 to 7333762027674202112. Current Snapshot: MvccSnapshot[applied={T|T < 7333762027946283008}]
I20260927 00:34:05.125584 16390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:05.125752 16390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:05.129796 16390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c76f503a46ff4540985caceb448ce3e7: Sending DeleteTablet for 3 replicas of tablet e9441cdf1c634e40b16570e4ee9eebd5
I20260927 00:34:05.130744 17431 tablet_service.cc:1514] Processing DeleteTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:05 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:34232
I20260927 00:34:05.130834 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:05 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:41176
I20260927 00:34:05.131578 17692 tablet_replica.cc:330] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: stopping tablet replica
I20260927 00:34:05.131954 17692 raft_consensus.cc:2229] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:05.132395 17692 raft_consensus.cc:2258] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:05.133203 17569 tablet_service.cc:1514] Processing DeleteTablet for tablet e9441cdf1c634e40b16570e4ee9eebd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:05 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:35600
I20260927 00:34:05.133445 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16359
I20260927 00:34:05.134537 17692 ts_tablet_manager.cc:1905] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:05.137586 17692 ts_tablet_manager.cc:1918] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:05.137722 17692 log.cc:1196] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/e9441cdf1c634e40b16570e4ee9eebd5
I20260927 00:34:05.138069 17692 ts_tablet_manager.cc:1939] T e9441cdf1c634e40b16570e4ee9eebd5 P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting consensus metadata
W20260927 00:34:05.140009 17384 connection.cc:535] server connection from 127.0.0.1:34232 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:34:05.141517 17695 tablet_replica.cc:330] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: stopping tablet replica
I20260927 00:34:05.141868 17695 raft_consensus.cc:2229] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:05.142234 17695 raft_consensus.cc:2258] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:05.142592 17693 tablet_replica.cc:330] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: stopping tablet replica
I20260927 00:34:05.142861 17693 raft_consensus.cc:2229] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:05.143496 17693 raft_consensus.cc:2258] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:05.144586 17695 ts_tablet_manager.cc:1905] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:05.147620 17695 ts_tablet_manager.cc:1918] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:05.147723 17695 log.cc:1196] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/e9441cdf1c634e40b16570e4ee9eebd5
I20260927 00:34:05.148108 17695 ts_tablet_manager.cc:1939] T e9441cdf1c634e40b16570e4ee9eebd5 P d294934e21a446bd8c8a872ca3266d0c: Deleting consensus metadata
W20260927 00:34:05.149281 17523 connection.cc:441] server connection from 127.0.0.1:35600 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35600 (request call id 1) could send its response
I20260927 00:34:05.151399 17693 ts_tablet_manager.cc:1905] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:05.153782 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.143092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.10.11.253:33761)
0927 00:34:05.143220 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.143225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.144229 (+ 1004us) server_negotiation.cc:407] Connection header received
0927 00:34:05.144385 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.144387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.144445 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.144494 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:05.145483 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.146266 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.147403 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.147784 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.147854 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.149977 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:05.149981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:05.149983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:05.150007 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:05.151677 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.152217 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.152221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.152223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.152300 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.152596 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.152599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.152601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.152805 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:05.152901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.153232 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.153352 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:34:05.156068 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.132484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.10.11.253:33761)
0927 00:34:05.132692 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.132698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.141206 (+ 8508us) server_negotiation.cc:407] Connection header received
0927 00:34:05.141361 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.141364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.141415 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.141488 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:05.142499 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.143215 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.144833 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.145141 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.145214 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.147344 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:05.147348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:05.147351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:05.147377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:05.149333 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.154208 (+ 4875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.154213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.154215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.154321 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.154649 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.154653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.154656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.154866 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:05.154987 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.155807 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.155914 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 00:34:05.156281 17693 ts_tablet_manager.cc:1918] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:05.156383 17693 log.cc:1196] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/e9441cdf1c634e40b16570e4ee9eebd5
I20260927 00:34:05.156723 17693 ts_tablet_manager.cc:1939] T e9441cdf1c634e40b16570e4ee9eebd5 P b83e20ead07c49c18bb6b753d04d350e: Deleting consensus metadata
W20260927 00:34:05.157845 17249 connection.cc:441] server connection from 127.0.0.1:41176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41176 (request call id 1) could send its response
I20260927 00:34:05.159335 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.143890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.10.11.252:34887)
0927 00:34:05.144001 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.144005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.149633 (+ 5628us) server_negotiation.cc:407] Connection header received
0927 00:34:05.149714 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.149717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.149781 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.149859 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:05.150805 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.151495 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.152662 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.153003 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.153078 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.154711 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:05.154714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:05.154717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:05.154755 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:05.157453 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.157996 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.157999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.158001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.158081 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.158407 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.158410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.158411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.158652 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:05.158787 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.159046 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.159157 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:34:05.164604 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:05.132576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.10.11.252:34887)
0927 00:34:05.132791 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:05.132796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:05.156058 (+ 23262us) server_negotiation.cc:407] Connection header received
0927 00:34:05.156336 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:05.156339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:05.156386 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:05.156472 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:05.157289 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.158070 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.159308 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:05.159669 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:05.159739 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:05.161277 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:05.161279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:05.161281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:05.161299 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:05.163190 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.163640 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.163643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.163644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.163709 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:05.163971 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:05.163973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:05.163974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:05.164114 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:05.164225 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:05.164421 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 00:34:05.164507 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260927 00:34:06.119092 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.118670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:59531)
0927 00:34:06.118898 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.118971 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
W20260927 00:34:06.119396 17635 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.120852 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.120346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:57163)
0927 00:34:06.120625 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.120722 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
W20260927 00:34:06.121232 17497 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.125291 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.124961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:50485)
0927 00:34:06.125156 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.125220 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
W20260927 00:34:06.125627 17360 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.630321 17706 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c76f503a46ff4540985caceb448ce3e7)
I20260927 00:34:06.630367 17707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c76f503a46ff4540985caceb448ce3e7)
I20260927 00:34:06.630508 17707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } }
I20260927 00:34:06.630523 17706 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } }
I20260927 00:34:06.631052 17707 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645), fa5126e3120f4e01bf7835377955f83e (127.10.11.253:33761)
I20260927 00:34:06.631588 17706 leader_election.cc:288] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645), 9cfd3fe29685423d97bebf11a63e99c8 (127.10.11.252:34887)
I20260927 00:34:06.632081 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.631672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57848)
0927 00:34:06.631903 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.631982 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
W20260927 00:34:06.632238 16976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.632362 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.631266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57842)
0927 00:34:06.632187 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.632265 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":73,"threads_started":1}
W20260927 00:34:06.632521 17166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.633379 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.633153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57850)
0927 00:34:06.633259 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.633317 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 00:34:06.633400 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.633163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57856)
0927 00:34:06.633300 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.633351 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:34:06.633502 16976 leader_election.cc:334] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645): Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
W20260927 00:34:06.633540 17166 leader_election.cc:334] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645): Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.635843 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.631277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:33761 (local address 127.0.0.1:44554)
0927 00:34:06.631586 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.631602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.631722 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.632132 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.632137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.632159 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.632258 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.632266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.633291 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.633294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.634739 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.634747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.635436 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.635439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.635539 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.635558 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.635642 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.635717 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":340,"thread_start_us":83,"threads_started":1}
I20260927 00:34:06.635856 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.631363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.10.11.253:33761)
0927 00:34:06.631621 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.631626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.631669 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:34:06.631854 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.631858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.631903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.631981 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.632389 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.633183 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.634849 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.635308 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.635424 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.635479 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.635655 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.635719 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":181,"thread_start_us":60,"threads_started":1}
I20260927 00:34:06.636262 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cfd3fe29685423d97bebf11a63e99c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa5126e3120f4e01bf7835377955f83e" is_pre_election: true
I20260927 00:34:06.636409 17000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cfd3fe29685423d97bebf11a63e99c8 in term 1.
I20260927 00:34:06.636457 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.631999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:34887 (local address 127.0.0.1:35432)
0927 00:34:06.632521 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.632537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.632668 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.632978 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.632981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.632996 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.633071 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.633075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.634489 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.634492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.635282 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.635291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.636069 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.636073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.636164 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.636189 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.636262 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.636313 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260927 00:34:06.636490 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.632108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.10.11.252:34887)
0927 00:34:06.632699 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.632752 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.632765 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:06.632814 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.632817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.632854 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.633158 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.633598 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.634380 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.635400 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.635815 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.636071 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.636158 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.636305 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.636378 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":48,"threads_started":1}
I20260927 00:34:06.636922 17167 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cfd3fe29685423d97bebf11a63e99c8, fa5126e3120f4e01bf7835377955f83e; no voters: c76f503a46ff4540985caceb448ce3e7
I20260927 00:34:06.637010 17191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa5126e3120f4e01bf7835377955f83e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cfd3fe29685423d97bebf11a63e99c8" is_pre_election: true
I20260927 00:34:06.637102 17707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:34:06.637156 17707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c76f503a46ff4540985caceb448ce3e7)
I20260927 00:34:06.637207 17707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:34:06.638314 17707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } }
I20260927 00:34:06.638597 17191 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa5126e3120f4e01bf7835377955f83e in current term 2: Already voted for candidate 9cfd3fe29685423d97bebf11a63e99c8 in this term.
I20260927 00:34:06.638697 17707 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 election: Requested vote from peers c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645), fa5126e3120f4e01bf7835377955f83e (127.10.11.253:33761)
I20260927 00:34:06.639148 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.638908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57866)
0927 00:34:06.639025 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.639079 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:34:06.639283 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cfd3fe29685423d97bebf11a63e99c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa5126e3120f4e01bf7835377955f83e"
I20260927 00:34:06.639328 16976 leader_election.cc:302] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa5126e3120f4e01bf7835377955f83e; no voters: 9cfd3fe29685423d97bebf11a63e99c8, c76f503a46ff4540985caceb448ce3e7
I20260927 00:34:06.639421 17000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:34:06.639815 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.639600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57880)
0927 00:34:06.639712 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.639764 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:34:06.639933 17166 leader_election.cc:334] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645): Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
I20260927 00:34:06.640515 17000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cfd3fe29685423d97bebf11a63e99c8 in term 2.
I20260927 00:34:06.640677 17706 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:34:06.640831 17167 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cfd3fe29685423d97bebf11a63e99c8, fa5126e3120f4e01bf7835377955f83e; no voters: c76f503a46ff4540985caceb448ce3e7
I20260927 00:34:06.641038 17707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:34:06.641266 17707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [term 2 LEADER]: Becoming Leader. State: Replica: 9cfd3fe29685423d97bebf11a63e99c8, State: Running, Role: LEADER
I20260927 00:34:06.641515 17707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } }
I20260927 00:34:06.642536 17716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfd3fe29685423d97bebf11a63e99c8. Latest consensus state: current_term: 2 leader_uuid: "9cfd3fe29685423d97bebf11a63e99c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:06.642663 17716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:06.642985 17718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:06.645028 17718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:06.645408 17718 catalog_manager.cc:1259] Loaded cluster ID: 72017ef12c144e48a44494858c9392df
I20260927 00:34:06.645473 17718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:06.645998 17718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:06.646410 17718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Loaded TSK: 0
I20260927 00:34:06.647042 17718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:34:06.668692 17181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:06.671686 17181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:06.674391 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.674141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57900)
0927 00:34:06.674280 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.674329 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:34:06.674434 17000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [term 2 FOLLOWER]: Refusing update from remote peer 9cfd3fe29685423d97bebf11a63e99c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 00:34:06.675050 17707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 00:34:06.675174 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.674933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.0.0.1:57910)
0927 00:34:06.675039 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.675086 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:34:06.675392 17166 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 -> Peer c76f503a46ff4540985caceb448ce3e7 (127.10.11.254:35645): Couldn't send request to peer c76f503a46ff4540985caceb448ce3e7. Status: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:34:06.676520 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfd3fe29685423d97bebf11a63e99c8. Latest consensus state: current_term: 2 leader_uuid: "9cfd3fe29685423d97bebf11a63e99c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:06.676656 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:06.677621 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9cfd3fe29685423d97bebf11a63e99c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:06.677747 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:06.678216 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9cfd3fe29685423d97bebf11a63e99c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c76f503a46ff4540985caceb448ce3e7" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35645 } } peers { permanent_uuid: "fa5126e3120f4e01bf7835377955f83e" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 33761 } attrs { promote: false } } peers { permanent_uuid: "9cfd3fe29685423d97bebf11a63e99c8" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 34887 } attrs { promote: false } } }
I20260927 00:34:06.678323 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5126e3120f4e01bf7835377955f83e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:34:06.687839 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.683210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:35729 (local address 127.0.0.1:34246)
0927 00:34:06.683371 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.683386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.683456 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.684171 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.684174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.684193 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.684306 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.684313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685493 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.685496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.686647 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.686655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687451 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.687556 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.687574 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.687618 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.687707 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":33}
I20260927 00:34:06.687839 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.683391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35921 (local address 127.0.0.1:35608)
0927 00:34:06.683504 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.683519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.683602 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.684084 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.684088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.684103 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.684390 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.684395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685251 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.685257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.686549 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.686559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687405 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.687513 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.687566 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.687640 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.687724 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":266}
I20260927 00:34:06.687889 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.683619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.10.11.195:35921)
0927 00:34:06.683829 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.683833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.683847 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:06.683912 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.683915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.683965 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.684243 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.684494 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685145 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.686685 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687285 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687382 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.687441 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.687673 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.687754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260927 00:34:06.688268 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.683725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:38955 (local address 127.0.0.1:41196)
0927 00:34:06.684392 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.684405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.684482 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.684935 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.684938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.684956 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.685032 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.685037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685857 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.685862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.686942 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.686950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687851 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.687944 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.687969 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.688039 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.688110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":66,"thread_start_us":94,"threads_started":1}
I20260927 00:34:06.688601 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.684054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.10.11.193:38955)
0927 00:34:06.684676 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.684681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.684693 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:06.684739 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.684742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.684784 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.684902 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.685143 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685752 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687067 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687635 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.687732 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.687795 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.688413 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.688487 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":70,"threads_started":1}
I20260927 00:34:06.688627 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.683313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.10.11.194:35729)
0927 00:34:06.683928 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.683933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.683947 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:06.683997 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.684000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.684045 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.684513 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.684628 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.685388 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.686774 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.687307 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.688348 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.688425 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.688463 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.688532 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":72,"threads_started":1}
I20260927 00:34:06.688591 17569 tablet_service.cc:1467] Processing CreateTablet for tablet c1181edc34a14c86819b58c0966e39ba (DEFAULT_TABLE table=test-table [id=accd10d029014f6c800778220a90e882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:06.688854 17569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1181edc34a14c86819b58c0966e39ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:06.689093 17431 tablet_service.cc:1467] Processing CreateTablet for tablet c1181edc34a14c86819b58c0966e39ba (DEFAULT_TABLE table=test-table [id=accd10d029014f6c800778220a90e882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:06.689353 17431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1181edc34a14c86819b58c0966e39ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:06.691452 17729 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Bootstrap starting.
I20260927 00:34:06.691736 17730 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Bootstrap starting.
I20260927 00:34:06.691905 17294 tablet_service.cc:1467] Processing CreateTablet for tablet c1181edc34a14c86819b58c0966e39ba (DEFAULT_TABLE table=test-table [id=accd10d029014f6c800778220a90e882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:06.692154 17294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1181edc34a14c86819b58c0966e39ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:06.692396 17729 tablet_bootstrap.cc:654] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:06.692781 17730 tablet_bootstrap.cc:654] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:06.693703 17729 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: No bootstrap required, opened a new log
I20260927 00:34:06.693783 17729 ts_tablet_manager.cc:1397] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:06.694146 17730 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: No bootstrap required, opened a new log
I20260927 00:34:06.694227 17730 ts_tablet_manager.cc:1397] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:34:06.694290 17729 raft_consensus.cc:348] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.694414 17729 raft_consensus.cc:374] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:06.694455 17729 raft_consensus.cc:729] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d294934e21a446bd8c8a872ca3266d0c, State: Initialized, Role: FOLLOWER
I20260927 00:34:06.694588 17729 consensus_queue.cc:260] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.694753 17733 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e: Bootstrap starting.
I20260927 00:34:06.694680 17730 raft_consensus.cc:348] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.694808 17730 raft_consensus.cc:374] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:06.694852 17730 raft_consensus.cc:729] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bf4552f23f4064a5ecbab3836a5fe3, State: Initialized, Role: FOLLOWER
I20260927 00:34:06.694947 17729 ts_tablet_manager.cc:1428] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:34:06.694969 17730 consensus_queue.cc:260] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.695272 17730 ts_tablet_manager.cc:1428] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:06.695963 17733 tablet_bootstrap.cc:654] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:06.696990 17733 tablet_bootstrap.cc:492] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e: No bootstrap required, opened a new log
I20260927 00:34:06.697079 17733 ts_tablet_manager.cc:1397] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:06.697365 17733 raft_consensus.cc:348] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.697446 17733 raft_consensus.cc:374] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:06.697467 17733 raft_consensus.cc:729] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83e20ead07c49c18bb6b753d04d350e, State: Initialized, Role: FOLLOWER
I20260927 00:34:06.697590 17733 consensus_queue.cc:260] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.697907 17733 ts_tablet_manager.cc:1428] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:06.698809 17636 heartbeater.cc:503] Master 127.10.11.252:34887 was elected leader, sending a full tablet report...
I20260927 00:34:06.698967 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.698278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:58573)
0927 00:34:06.698821 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.698881 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":101,"threads_started":1}
I20260927 00:34:06.699770 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.699509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:58569)
0927 00:34:06.699633 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.699694 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:34:06.700577 17361 heartbeater.cc:503] Master 127.10.11.252:34887 was elected leader, sending a full tablet report...
I20260927 00:34:06.700666 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.695778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:38199)
0927 00:34:06.700501 (+ 4723us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.700552 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4662,"thread_start_us":96,"threads_started":1}
I20260927 00:34:06.701134 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.696112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:43451)
0927 00:34:06.700979 (+ 4867us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.701053 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4792,"thread_start_us":83,"threads_started":1}
I20260927 00:34:06.701604 17498 heartbeater.cc:503] Master 127.10.11.252:34887 was elected leader, sending a full tablet report...
I20260927 00:34:06.702244 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.701983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:37379)
0927 00:34:06.702070 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.702172 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 00:34:06.793706 17735 raft_consensus.cc:482] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:06.793882 17735 raft_consensus.cc:504] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.795004 17735 leader_election.cc:288] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921), b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
I20260927 00:34:06.800334 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.795383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:52071 (local address 127.10.11.195:35921)
0927 00:34:06.795535 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.795541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.795804 (+ 263us) server_negotiation.cc:407] Connection header received
0927 00:34:06.795881 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.795884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.795927 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.795998 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.796223 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.797027 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.798967 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.799582 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.799710 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.799763 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.800120 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.800204 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:34:06.800334 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.795532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40815 (local address 127.10.11.193:38955)
0927 00:34:06.795660 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.795666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.795684 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:34:06.795728 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.795730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.795803 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.795882 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:06.796223 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.797001 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.798967 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.799582 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.799710 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.799763 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.800120 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.800204 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:34:06.800364 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.795260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:38955 (local address 127.10.11.194:40815)
0927 00:34:06.795415 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.795431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.795543 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.795926 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.795929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.795942 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.796086 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.796095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.797137 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.797141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.798819 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.798829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.799759 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.799763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.799936 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.799979 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.800067 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.800203 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":278}
I20260927 00:34:06.800372 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.795260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:35921 (local address 127.10.11.194:52071)
0927 00:34:06.795700 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.795710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:06.795800 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:06.796020 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:06.796022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:06.796034 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:06.796115 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.796123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.797137 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.797141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.798817 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:06.798829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.799759 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.799763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:06.799894 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.799981 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:34:06.800067 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:34:06.800205 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":383,"thread_start_us":125,"threads_started":1}
I20260927 00:34:06.800957 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1181edc34a14c86819b58c0966e39ba" candidate_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83e20ead07c49c18bb6b753d04d350e" is_pre_election: true
I20260927 00:34:06.801178 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1181edc34a14c86819b58c0966e39ba" candidate_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d294934e21a446bd8c8a872ca3266d0c" is_pre_election: true
I20260927 00:34:06.801222 17314 raft_consensus.cc:2454] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4bf4552f23f4064a5ecbab3836a5fe3 in term 0.
I20260927 00:34:06.801301 17589 raft_consensus.cc:2454] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4bf4552f23f4064a5ecbab3836a5fe3 in term 0.
I20260927 00:34:06.801669 17384 leader_election.cc:302] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b83e20ead07c49c18bb6b753d04d350e, d294934e21a446bd8c8a872ca3266d0c, d4bf4552f23f4064a5ecbab3836a5fe3; no voters:
I20260927 00:34:06.802175 17735 raft_consensus.cc:2781] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:06.802242 17735 raft_consensus.cc:482] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:06.802295 17735 raft_consensus.cc:3037] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:06.803431 17735 raft_consensus.cc:504] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.803772 17735 leader_election.cc:288] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [CANDIDATE]: Term 1 election: Requested vote from peers d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921), b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955)
I20260927 00:34:06.804103 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1181edc34a14c86819b58c0966e39ba" candidate_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d294934e21a446bd8c8a872ca3266d0c"
I20260927 00:34:06.804117 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1181edc34a14c86819b58c0966e39ba" candidate_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83e20ead07c49c18bb6b753d04d350e"
I20260927 00:34:06.804190 17589 raft_consensus.cc:3037] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:06.804262 17314 raft_consensus.cc:3037] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:06.805076 17589 raft_consensus.cc:2454] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4bf4552f23f4064a5ecbab3836a5fe3 in term 1.
I20260927 00:34:06.805287 17314 raft_consensus.cc:2454] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4bf4552f23f4064a5ecbab3836a5fe3 in term 1.
I20260927 00:34:06.805375 17386 leader_election.cc:302] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d294934e21a446bd8c8a872ca3266d0c, d4bf4552f23f4064a5ecbab3836a5fe3; no voters:
I20260927 00:34:06.805554 17735 raft_consensus.cc:2781] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:06.805773 17735 raft_consensus.cc:686] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 LEADER]: Becoming Leader. State: Replica: d4bf4552f23f4064a5ecbab3836a5fe3, State: Running, Role: LEADER
I20260927 00:34:06.806018 17735 consensus_queue.cc:237] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } }
I20260927 00:34:06.806504 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.806258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:60255)
0927 00:34:06.806370 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.806434 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 00:34:06.808518 17181 catalog_manager.cc:5495] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bf4552f23f4064a5ecbab3836a5fe3" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20260927 00:34:06.840109 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.836701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.10.11.194:35729)
0927 00:34:06.836833 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:06.836837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:06.836851 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:06.836998 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:06.837000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:06.837053 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:06.837143 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:06.837318 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.837944 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.838746 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:06.838980 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:06.839108 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:06.839301 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:06.839718 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:06.839801 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:06.839903 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:34:06.839948 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:34:06.842693 17589 raft_consensus.cc:1261] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Refusing update from remote peer d4bf4552f23f4064a5ecbab3836a5fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:06.842979 17314 raft_consensus.cc:1261] T c1181edc34a14c86819b58c0966e39ba P b83e20ead07c49c18bb6b753d04d350e [term 1 FOLLOWER]: Refusing update from remote peer d4bf4552f23f4064a5ecbab3836a5fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:06.843175 17735 consensus_queue.cc:1035] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d294934e21a446bd8c8a872ca3266d0c" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:06.843554 17742 consensus_queue.cc:1035] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83e20ead07c49c18bb6b753d04d350e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:06.844831 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.844510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:36129)
0927 00:34:06.844708 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.844775 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 00:34:06.846168 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.845890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:40373)
0927 00:34:06.846019 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.846075 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:34:06.848239 17746 mvcc.cc:205] Tried to move back new op lower bound from 7333762035063050240 to 7333762034920300544. Current Snapshot: MvccSnapshot[applied={T|T < 7333762035063050240}]
I20260927 00:34:06.848960 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.848644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.195:37651)
0927 00:34:06.848851 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.848900 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 00:34:06.850935 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.850395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:45625)
0927 00:34:06.850806 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.850858 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260927 00:34:06.852065 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.851846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.193:59133)
0927 00:34:06.851952 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.852000 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:34:06.852207 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.851992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35645 (local address 127.10.11.194:46989)
0927 00:34:06.852111 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.852152 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 00:34:06.856745 17180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:06.856906 17180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:06.860646 17180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cfd3fe29685423d97bebf11a63e99c8: Sending DeleteTablet for 3 replicas of tablet c1181edc34a14c86819b58c0966e39ba
I20260927 00:34:06.861390 17569 tablet_service.cc:1514] Processing DeleteTablet for tablet c1181edc34a14c86819b58c0966e39ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:06 UTC) from {username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:35608
I20260927 00:34:06.861995 17431 tablet_service.cc:1514] Processing DeleteTablet for tablet c1181edc34a14c86819b58c0966e39ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:06 UTC) from {username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:34246
I20260927 00:34:06.862174 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet c1181edc34a14c86819b58c0966e39ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:06 UTC) from {username='oryx', principal='oryx/127.10.11.252@KRBTEST.COM'} at 127.0.0.1:41196
I20260927 00:34:06.862381 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 17136
I20260927 00:34:06.864200 17756 tablet_replica.cc:330] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: stopping tablet replica
I20260927 00:34:06.864363 17756 raft_consensus.cc:2229] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:06.864673 17756 raft_consensus.cc:2258] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:06.864907 17755 tablet_replica.cc:330] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: stopping tablet replica
I20260927 00:34:06.865185 17755 raft_consensus.cc:2229] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:06.865331 17755 raft_consensus.cc:2258] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:06.865559 17756 ts_tablet_manager.cc:1905] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:06.866003 17755 ts_tablet_manager.cc:1905] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:06.868153 17756 ts_tablet_manager.cc:1918] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:06.868232 17756 log.cc:1196] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c1181edc34a14c86819b58c0966e39ba
I20260927 00:34:06.868530 17756 ts_tablet_manager.cc:1939] T c1181edc34a14c86819b58c0966e39ba P d4bf4552f23f4064a5ecbab3836a5fe3: Deleting consensus metadata
I20260927 00:34:06.869132 17165 catalog_manager.cc:4841] TS d4bf4552f23f4064a5ecbab3836a5fe3 (127.10.11.194:35729): tablet c1181edc34a14c86819b58c0966e39ba (table test-table [id=accd10d029014f6c800778220a90e882]) successfully deleted
I20260927 00:34:06.870822 17755 ts_tablet_manager.cc:1918] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:06.871068 17755 log.cc:1196] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c1181edc34a14c86819b58c0966e39ba
I20260927 00:34:06.871272 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 17368
I20260927 00:34:06.871553 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:06.871304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:38955 (local address 127.0.0.1:41198)
0927 00:34:06.871450 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:06.871491 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:38955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 00:34:06.872299 17755 ts_tablet_manager.cc:1939] T c1181edc34a14c86819b58c0966e39ba P d294934e21a446bd8c8a872ca3266d0c: Deleting consensus metadata
W20260927 00:34:06.872503 17165 catalog_manager.cc:4584] TS b83e20ead07c49c18bb6b753d04d350e (127.10.11.193:38955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1181edc34a14c86819b58c0966e39ba: Network error: Client connection negotiation failed: client connection to 127.10.11.193:38955: connect: Connection refused (error 111)
I20260927 00:34:06.872964 17167 catalog_manager.cc:4841] TS d294934e21a446bd8c8a872ca3266d0c (127.10.11.195:35921): tablet c1181edc34a14c86819b58c0966e39ba (table test-table [id=accd10d029014f6c800778220a90e882]) successfully deleted
I20260927 00:34:06.876945 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 17504
I20260927 00:34:06.882560 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16855
I20260927 00:34:06.888186 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 16957
2026-09-27T00:34:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5388 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:06 dist-test-slave-6gxc krb5kdc[17760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:06 dist-test-slave-6gxc krb5kdc[17760](info): set up 2 sockets
Sep 27 00:34:06 dist-test-slave-6gxc krb5kdc[17760](info): commencing operation
krb5kdc: starting...
W20260927 00:34:07.918927 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:07 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab.
Entry for principal oryx/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab.
I20260927 00:34:08.033365 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38255
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab
--principal=oryx/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:38255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:08.137809 17773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:08.138054 17773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:08.138142 17773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:08.142095 17773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:08.142184 17773 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:34:08.142211 17773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:08.142249 17773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:08.142283 17773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:08.142316 17773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:08.147383 17773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:38255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.254.keytab
--principal=oryx/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:08.148304 17773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:08.149201 17773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:08.156952 17781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.156962 17778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.156962 17779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:08.157069 17773 server_base.cc:1034] running on GCE node
I20260927 00:34:08.157733 17773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:08.158776 17773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:08.160004 17773 hybrid_clock.cc:648] HybridClock initialized: now 1790469248159975 us; error 44 us; skew 500 ppm
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:08.162604 17773 init.cc:377] Logged in from keytab as oryx/127.10.11.254@KRBTEST.COM (short username oryx)
I20260927 00:34:08.163480 17773 webserver.cc:460] Webserver started at http://127.10.11.254:36115/ using document root <none> and password file <none>
I20260927 00:34:08.164018 17773 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:08.164103 17773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:08.164323 17773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:08.166270 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "beab0ed945a24b8a80db413e4eeaa9a6"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "d315c55e1b179657ceae6dac9b0feb8e"
server_key_iv: "6bd311b32ef8355dadc3317fac691399"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.166792 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "beab0ed945a24b8a80db413e4eeaa9a6"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "d315c55e1b179657ceae6dac9b0feb8e"
server_key_iv: "6bd311b32ef8355dadc3317fac691399"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.170197 17773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:08.172402 17788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.173560 17773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:08.173707 17773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "beab0ed945a24b8a80db413e4eeaa9a6"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "d315c55e1b179657ceae6dac9b0feb8e"
server_key_iv: "6bd311b32ef8355dadc3317fac691399"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.173807 17773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:08.194875 17773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:08.197420 17773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:08.197605 17773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:08.205101 17773 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:38255
I20260927 00:34:08.205123 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:38255 every 8 connection(s)
I20260927 00:34:08.205940 17773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:08.208303 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 17773
I20260927 00:34:08.208408 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:08.208477 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:08.208716 10287 external_mini_cluster.cc:1428] Setting key f93fef74313dbc7de4844786b125c1a4
I20260927 00:34:08.215147 17851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Bootstrap starting.
I20260927 00:34:08.217417 17851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Neither blocks nor log segments found. Creating new log.
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:08.218156 17851 log.cc:824] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:08.220307 17851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: No bootstrap required, opened a new log
I20260927 00:34:08.223946 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.210066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.10.11.254:38255)
0927 00:34:08.210506 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.210523 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.210564 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:34:08.211705 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.211729 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.212121 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.212469 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.213109 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.214569 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.215424 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.216052 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.216210 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.218621 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.218626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.218636 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.218659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.221000 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.221420 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.221426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.221431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.221559 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.221894 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.221900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.221902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.222386 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.222569 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.222800 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.223072 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":116,"threads_started":1}
I20260927 00:34:08.224028 17851 raft_consensus.cc:348] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } }
I20260927 00:34:08.224274 17851 raft_consensus.cc:374] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:08.224336 17851 raft_consensus.cc:729] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beab0ed945a24b8a80db413e4eeaa9a6, State: Initialized, Role: FOLLOWER
I20260927 00:34:08.224748 17851 consensus_queue.cc:260] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } }
I20260927 00:34:08.224910 17851 raft_consensus.cc:388] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:08.224974 17851 raft_consensus.cc:482] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:08.225068 17851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:08.225898 17851 raft_consensus.cc:504] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } }
I20260927 00:34:08.226253 17851 leader_election.cc:302] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beab0ed945a24b8a80db413e4eeaa9a6; no voters:
I20260927 00:34:08.226521 17851 leader_election.cc:288] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:08.226625 17856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:08.226845 17856 raft_consensus.cc:686] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [term 1 LEADER]: Becoming Leader. State: Replica: beab0ed945a24b8a80db413e4eeaa9a6, State: Running, Role: LEADER
I20260927 00:34:08.227355 17856 consensus_queue.cc:237] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } }
I20260927 00:34:08.227696 17851 sys_catalog.cc:564] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:08.229094 17857 sys_catalog.cc:455] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } } }
I20260927 00:34:08.229138 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beab0ed945a24b8a80db413e4eeaa9a6. Latest consensus state: current_term: 1 leader_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab0ed945a24b8a80db413e4eeaa9a6" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38255 } } }
I20260927 00:34:08.229223 17857 sys_catalog.cc:458] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:08.229229 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6 [sys.catalog]: This master's current role is: LEADER
W20260927 00:34:08.232208 17871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:34:08.232270 17871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:34:08.232344 17869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:08.233078 17869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:08.237869 17869 catalog_manager.cc:1347] Generated new cluster ID: b4f097ce93f34f19b7345fc7566d052c
I20260927 00:34:08.237926 17869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:08.258806 17869 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:08.259917 17869 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:08.271787 17869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Generated new TSK 0
I20260927 00:34:08.272491 17869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
I20260927 00:34:08.307096 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38255
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:08.411983 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:08.412196 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:08.412236 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:08.416144 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:08.416191 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:08.416227 17877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:34:08.416294 17877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:08.421149 17877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:38255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:08.422017 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:08.422951 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:08.431983 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.432008 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.432008 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:08.432106 17877 server_base.cc:1034] running on GCE node
I20260927 00:34:08.432560 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:08.433135 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:08.434316 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790469248434300 us; error 38 us; skew 500 ppm
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:08.436998 17877 init.cc:377] Logged in from keytab as oryx/127.10.11.193@KRBTEST.COM (short username oryx)
I20260927 00:34:08.437855 17877 webserver.cc:460] Webserver started at http://127.10.11.193:38293/ using document root <none> and password file <none>
I20260927 00:34:08.438398 17877 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:08.438459 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:08.438608 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:08.440471 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "9b34a01773569e290bc559de8d34a118"
server_key_iv: "3c1c12debf4f3ae4896ed4a7080d5cec"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.440960 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "9b34a01773569e290bc559de8d34a118"
server_key_iv: "3c1c12debf4f3ae4896ed4a7080d5cec"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.444521 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:34:08.446770 17892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.447919 17877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:08.448007 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "9b34a01773569e290bc559de8d34a118"
server_key_iv: "3c1c12debf4f3ae4896ed4a7080d5cec"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.448086 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:08.466521 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:08.469451 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:08.469587 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:08.470916 17877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:08.470989 17877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.471052 17877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:08.471091 17877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:08.481523 17877 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:34325
I20260927 00:34:08.481602 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:34325 every 8 connection(s)
I20260927 00:34:08.482504 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:08.483073 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 17877
I20260927 00:34:08.483219 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:08.483542 10287 external_mini_cluster.cc:1428] Setting key b11e8a3d597cb40321ef73f4a71e8b32
I20260927 00:34:08.484453 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.471525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:60999 (local address 127.10.11.254:38255)
0927 00:34:08.471658 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.471663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.472543 (+ 880us) server_negotiation.cc:407] Connection header received
0927 00:34:08.473557 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.473561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.473612 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.473664 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.474991 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.475664 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.476920 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.477288 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.477401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.480728 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.480731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.480733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.480762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.482327 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.482882 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.482886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.482887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.482975 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.483381 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.483384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.483385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.483568 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.483678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.484171 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.484287 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:34:08.485201 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.471954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.193:60999)
0927 00:34:08.472387 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.472433 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.473320 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.474051 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.474061 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.474421 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.474843 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.474855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.475807 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.475810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.476802 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.476811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.477457 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.477461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.477677 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.478298 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.478317 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.480556 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.482429 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.482436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.482449 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.482757 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.483084 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.483087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.483089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.483242 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.483676 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.483682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.483936 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.484125 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.484420 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260927 00:34:08.486546 18008 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38255
I20260927 00:34:08.486697 18008 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:08.487216 18008 heartbeater.cc:511] Master 127.10.11.254:38255 requested a full tablet report, sending...
I20260927 00:34:08.488724 17804 ts_manager.cc:194] Registered new tserver with Master: 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325)
I20260927 00:34:08.489758 17804 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:60999
WARNING: no policy specified for oryx/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.194@KRBTEST.COM" created.
I20260927 00:34:08.497493 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.490457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:33143 (local address 127.10.11.254:38255)
0927 00:34:08.490563 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.490567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.490829 (+ 262us) server_negotiation.cc:407] Connection header received
0927 00:34:08.490975 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.490978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.491016 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.491107 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.491506 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.491983 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.492768 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.493033 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.493146 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.494293 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.494297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.494300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.494322 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.495731 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.496329 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.496335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.496337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.496419 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.496698 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.496703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.496704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.496895 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.496999 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.497255 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.497351 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:34:08.497525 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.490411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.193:33143)
0927 00:34:08.490744 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.490755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.490883 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.491121 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.491124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.491312 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.491396 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.491401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.492104 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.492107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.492652 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.492659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.493147 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.493150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.493243 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.493674 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.493684 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.494163 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.495857 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.495863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.495866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.496057 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.496518 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.496521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.496523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.496609 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.496999 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.497003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.497092 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.497194 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.497342 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":196,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
I20260927 00:34:08.509330 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38255
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:08.613987 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:08.614225 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:08.614266 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:08.618067 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:08.618129 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:08.618165 18017 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:34:08.618196 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:08.623078 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:38255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:08.623975 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:08.624846 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:08.634002 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.634040 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.634148 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:08.634162 18017 server_base.cc:1034] running on GCE node
I20260927 00:34:08.634691 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:08.635360 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:08.636550 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790469248636531 us; error 30 us; skew 500 ppm
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:08.639253 18017 init.cc:377] Logged in from keytab as oryx/127.10.11.194@KRBTEST.COM (short username oryx)
I20260927 00:34:08.640026 18017 webserver.cc:460] Webserver started at http://127.10.11.194:35161/ using document root <none> and password file <none>
I20260927 00:34:08.640539 18017 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:08.640626 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:08.640846 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:08.642730 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "1ccc69c039a046959ecf8e5de45941b4"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "8dcb7a11952281bbec2e139594dcf6cd"
server_key_iv: "7c2f080bb2c4f497c1dfd59cf3eaf331"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.643353 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "1ccc69c039a046959ecf8e5de45941b4"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "8dcb7a11952281bbec2e139594dcf6cd"
server_key_iv: "7c2f080bb2c4f497c1dfd59cf3eaf331"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.646790 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:08.649025 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.650179 18017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:08.650296 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "1ccc69c039a046959ecf8e5de45941b4"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "8dcb7a11952281bbec2e139594dcf6cd"
server_key_iv: "7c2f080bb2c4f497c1dfd59cf3eaf331"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.650415 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:08.663427 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:08.665884 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:08.666062 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:08.667337 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:08.667408 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.667477 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:08.667534 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:08.677772 18017 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:33071
I20260927 00:34:08.677863 18147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:33071 every 8 connection(s)
I20260927 00:34:08.678766 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:08.681805 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.668210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:38295 (local address 127.10.11.254:38255)
0927 00:34:08.668333 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.668337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.669277 (+ 940us) server_negotiation.cc:407] Connection header received
0927 00:34:08.670339 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.670346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.670414 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.670480 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.672138 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.673020 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.674957 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.675298 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.675431 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.678410 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.678414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.678418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.678449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.679999 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.680487 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.680489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.680491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.680561 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.680916 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.680918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.680919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.681070 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.681141 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.681602 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.681694 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:34:08.682591 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.668696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.194:38295)
0927 00:34:08.669130 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.669166 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.670075 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.670950 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.670962 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.671366 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.671989 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.672011 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.673169 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.673173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.674821 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.674831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.675461 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.675466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.675661 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.676351 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.676377 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.678230 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.680104 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.680108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.680126 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.680388 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.680664 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.680667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.680668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.680795 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.681185 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.681190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.681387 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.681531 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.681757 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
I20260927 00:34:08.683862 18148 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38255
I20260927 00:34:08.684006 18148 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:08.684430 18148 heartbeater.cc:511] Master 127.10.11.254:38255 requested a full tablet report, sending...
I20260927 00:34:08.684760 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18017
I20260927 00:34:08.684900 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:08.685228 10287 external_mini_cluster.cc:1428] Setting key a7e1503bbf08ab91c60439bfbef6dce7
I20260927 00:34:08.685398 17804 ts_manager.cc:194] Registered new tserver with Master: 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071)
I20260927 00:34:08.686102 17804 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:38295
I20260927 00:34:08.693581 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.685973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.194:48783)
0927 00:34:08.686317 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.686331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.686451 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.686814 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.686816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.686972 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.687056 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.687061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.687855 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.687859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.688640 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.688653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.689080 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.689083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.689209 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.689708 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.689719 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.690015 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.692141 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.692143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.692144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.692334 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.692636 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.692638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.692640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.692715 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.693203 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.693205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.693275 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.693364 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.693453 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260927 00:34:08.693718 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.686035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:48783 (local address 127.10.11.254:38255)
0927 00:34:08.686160 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.686164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.686387 (+ 223us) server_negotiation.cc:407] Connection header received
0927 00:34:08.686573 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.686576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.686618 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.686697 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.687169 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.687732 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.688765 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.688969 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.689050 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.690179 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.690185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.690191 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.690222 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.692015 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.692431 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.692437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.692442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.692533 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.692811 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.692815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.692819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.693099 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.693204 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.693447 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.693553 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
I20260927 00:34:08.713542 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38255
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:08.818392 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:08.818627 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:08.818699 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:08.822575 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:08.822620 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:08.822657 18157 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:34:08.822729 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:08.827643 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.11.254:38255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:08.828639 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:08.829608 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:08.838819 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.838825 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:08.838826 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:08.839182 18157 server_base.cc:1034] running on GCE node
I20260927 00:34:08.839569 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:08.840144 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:08.841351 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790469248841327 us; error 49 us; skew 500 ppm
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:08.844046 18157 init.cc:377] Logged in from keytab as oryx/127.10.11.195@KRBTEST.COM (short username oryx)
I20260927 00:34:08.844817 18157 webserver.cc:460] Webserver started at http://127.10.11.195:37107/ using document root <none> and password file <none>
I20260927 00:34:08.845379 18157 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:08.845472 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:08.845691 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:08.847743 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "30d76496879a7635c88a06d37b80c12a"
server_key_iv: "3041da952e55ce588b1bb8eee91650e1"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.848305 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "30d76496879a7635c88a06d37b80c12a"
server_key_iv: "3041da952e55ce588b1bb8eee91650e1"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.851850 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:34:08.854329 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.855482 18157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:08.855618 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
format_stamp: "Formatted at 2026-09-27 00:34:08 on dist-test-slave-6gxc"
server_key: "30d76496879a7635c88a06d37b80c12a"
server_key_iv: "3041da952e55ce588b1bb8eee91650e1"
server_key_version: "encryptionkey@0"
I20260927 00:34:08.855720 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:08.869750 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:08.872426 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:08.872613 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:08.873867 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:08.873937 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:08.874006 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:08.874055 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:08.884721 18157 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:42857
I20260927 00:34:08.884951 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:42857 every 8 connection(s)
I20260927 00:34:08.885635 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:08.887645 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.874727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:46451 (local address 127.10.11.254:38255)
0927 00:34:08.874904 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.874910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.875663 (+ 753us) server_negotiation.cc:407] Connection header received
0927 00:34:08.876631 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.876638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.876717 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.876801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.878592 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.879188 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.880411 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.880813 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.880962 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.883751 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.883755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.883757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.883785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.885473 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.886032 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.886036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.886038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.886159 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.886532 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.886537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.886539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.886720 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.886870 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.887344 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.887467 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:34:08.888304 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.875085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.195:46451)
0927 00:34:08.875509 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.875553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.876390 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.877263 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.877271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.877723 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.878421 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.878441 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.879375 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.879380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.880303 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.880311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.880990 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.880995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.881221 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.881814 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.881836 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.883582 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.885588 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.885593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.885606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.885922 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.886263 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.886266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.886268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.886421 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.886887 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.886894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.887112 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.887298 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.887592 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
I20260927 00:34:08.889045 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18157
I20260927 00:34:08.889163 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:08.889483 10287 external_mini_cluster.cc:1428] Setting key 1afd4ebcadb05c1fe2a02cf951aaeb00
I20260927 00:34:08.889528 18288 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38255
I20260927 00:34:08.889653 18288 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:08.890210 18288 heartbeater.cc:511] Master 127.10.11.254:38255 requested a full tablet report, sending...
I20260927 00:34:08.891213 17805 ts_manager.cc:194] Registered new tserver with Master: 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:08.892067 17805 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:46451
I20260927 00:34:08.892195 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:34:08.899577 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.891682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:43907 (local address 127.10.11.254:38255)
0927 00:34:08.891796 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.891800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.892135 (+ 335us) server_negotiation.cc:407] Connection header received
0927 00:34:08.892275 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.892278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.892320 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.892395 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.892996 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.893795 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.894833 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.895066 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.895175 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.896243 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.896246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.896249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.896272 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.897881 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.898415 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.898419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.898420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.898503 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.898786 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.898791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.898822 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.898983 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.899093 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.899346 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.899448 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:34:08.899592 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.891776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.10.11.195:43907)
0927 00:34:08.892039 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.892052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.892180 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.892506 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.892509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.892716 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:08.892855 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.892863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.893957 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.893963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.894715 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.894724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.895177 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.895180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.895269 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.895747 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:08.895758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:08.896055 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:08.898006 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.898009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.898010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.898278 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.898605 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:08.898607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:08.898609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:08.898696 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:08.899083 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:08.899086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:08.899171 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.899283 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.899418 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:08.915391 17854 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:08.909447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.10.11.254:38255)
0927 00:34:08.909552 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.909556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.909574 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:34:08.909603 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.909605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.909635 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.909702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.910570 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.911077 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.911941 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.912139 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.912219 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.915250 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33542: BlockingRecv error: recv got EOF from 127.0.0.1:33542 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 00:34:08 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:08.923959 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.916639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.10.11.254:38255)
0927 00:34:08.916749 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.916752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.916923 (+ 171us) server_negotiation.cc:407] Connection header received
0927 00:34:08.917005 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.917007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.917039 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.917103 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:08.917632 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.918087 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.918863 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.919060 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.919132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.921255 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:08.921258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:08.921260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:08.921279 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:08.922469 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.922855 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.922859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.922861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.922936 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:08.923198 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:08.923203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:08.923205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:08.923356 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:08.923496 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.923717 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.923819 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:34:08.926429 17805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:08.929394 17805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:08.943169 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.937501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:33071 (local address 127.0.0.1:49810)
0927 00:34:08.938555 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.938567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.938654 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.939224 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.939232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.939259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:08.939438 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.939446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.940447 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.940450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.941594 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.941602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.942694 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.942697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.942770 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.942795 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.942923 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.943037 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":31,"thread_start_us":95,"threads_started":1}
I20260927 00:34:08.943179 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.938173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.10.11.194:33071)
0927 00:34:08.938569 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.938581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.938775 (+ 194us) server_negotiation.cc:407] Connection header received
0927 00:34:08.938840 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.938844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.939057 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.939210 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:08.939561 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.940324 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.941716 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.942417 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.942517 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.942645 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.942950 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.943040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":96,"threads_started":1}
I20260927 00:34:08.943974 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.937745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34325 (local address 127.0.0.1:45668)
0927 00:34:08.938819 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.938872 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.938958 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.939367 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.939370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.939388 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:08.939589 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.939599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.941013 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.941017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.942290 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.942298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.943570 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.943574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.943671 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.943689 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.943775 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.943852 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":194,"thread_start_us":81,"threads_started":1}
I20260927 00:34:08.943984 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.937527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.10.11.195:42857)
0927 00:34:08.938840 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.938852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.938875 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:34:08.938951 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.938956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.939161 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.939447 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:08.939769 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.940882 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.942503 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.943220 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.943372 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.943572 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.943769 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.943853 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":120,"threads_started":1}
I20260927 00:34:08.944423 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.937393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42857 (local address 127.0.0.1:44618)
0927 00:34:08.938187 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:08.938234 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:08.938343 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:08.939306 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:08.939309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:08.939321 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:08.939633 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.939637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.941013 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.941017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.942375 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:08.942387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.943496 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.943503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:08.943628 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.943650 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:34:08.943723 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:34:08.944291 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":229,"thread_start_us":69,"threads_started":1}
I20260927 00:34:08.944576 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:08.938175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.10.11.193:34325)
0927 00:34:08.938735 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:08.938743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:08.938962 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:34:08.939067 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:08.939073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:08.939236 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:08.939367 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:08.939720 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.940885 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.942420 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:08.943221 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:08.943335 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:08.944267 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:08.944320 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:34:08.944408 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":100,"threads_started":1}
I20260927 00:34:08.946357 18082 tablet_service.cc:1467] Processing CreateTablet for tablet d0b968d76b434735afc51e525afc8893 (DEFAULT_TABLE table=test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:08.947242 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b968d76b434735afc51e525afc8893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:08.947037 18214 tablet_service.cc:1467] Processing CreateTablet for tablet d0b968d76b434735afc51e525afc8893 (DEFAULT_TABLE table=test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:08.947036 17942 tablet_service.cc:1467] Processing CreateTablet for tablet d0b968d76b434735afc51e525afc8893 (DEFAULT_TABLE table=test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:08.947973 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b968d76b434735afc51e525afc8893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:08.947983 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b968d76b434735afc51e525afc8893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:08.954092 18316 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Bootstrap starting.
I20260927 00:34:08.955943 18318 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Bootstrap starting.
I20260927 00:34:08.956739 18316 tablet_bootstrap.cc:654] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:08.957419 18316 log.cc:824] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:08.957594 18318 tablet_bootstrap.cc:654] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:08.958309 18318 log.cc:824] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:08.958405 18317 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Bootstrap starting.
I20260927 00:34:08.959301 18316 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: No bootstrap required, opened a new log
I20260927 00:34:08.959484 18316 ts_tablet_manager.cc:1397] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 00:34:08.960322 18318 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: No bootstrap required, opened a new log
I20260927 00:34:08.960421 18317 tablet_bootstrap.cc:654] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:08.960500 18318 ts_tablet_manager.cc:1397] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:34:08.961170 18317 log.cc:824] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:08.963020 18317 tablet_bootstrap.cc:492] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: No bootstrap required, opened a new log
I20260927 00:34:08.963186 18317 ts_tablet_manager.cc:1397] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:34:08.963119 18316 raft_consensus.cc:348] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.963404 18316 raft_consensus.cc:374] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:08.963459 18316 raft_consensus.cc:729] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Initialized, Role: FOLLOWER
I20260927 00:34:08.963883 18316 consensus_queue.cc:260] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.964244 18318 raft_consensus.cc:348] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.964491 18318 raft_consensus.cc:374] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:08.964502 18148 heartbeater.cc:503] Master 127.10.11.254:38255 was elected leader, sending a full tablet report...
I20260927 00:34:08.964541 18318 raft_consensus.cc:729] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc765f76eb24ce1b975ef89a9660e0d, State: Initialized, Role: FOLLOWER
I20260927 00:34:08.964929 18318 consensus_queue.cc:260] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.965538 18288 heartbeater.cc:503] Master 127.10.11.254:38255 was elected leader, sending a full tablet report...
I20260927 00:34:08.965866 18318 ts_tablet_manager.cc:1428] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:34:08.966852 18317 raft_consensus.cc:348] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.967070 18317 raft_consensus.cc:374] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:08.967128 18317 raft_consensus.cc:729] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Initialized, Role: FOLLOWER
I20260927 00:34:08.967510 18317 consensus_queue.cc:260] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:08.968540 18317 ts_tablet_manager.cc:1428] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:34:08.968837 18008 heartbeater.cc:503] Master 127.10.11.254:38255 was elected leader, sending a full tablet report...
I20260927 00:34:08.970824 18316 ts_tablet_manager.cc:1428] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260927 00:34:08.983883 18009 tablet.cc:2369] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:09.003942 18322 raft_consensus.cc:482] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.004104 18322 raft_consensus.cc:504] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:09.005152 18322 leader_election.cc:288] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857), 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325)
I20260927 00:34:09.008757 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.005389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42857 (local address 127.10.11.194:45669)
0927 00:34:09.005507 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.005523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.005605 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.005922 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.005924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.005940 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:09.006030 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.006035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.006819 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.006823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.007626 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.007634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.008276 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.008280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.008455 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.008473 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.008561 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.008628 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":54}
I20260927 00:34:09.008757 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.005480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:45669 (local address 127.10.11.195:42857)
0927 00:34:09.005584 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.005587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.005600 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:09.005714 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.005716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.005754 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.005879 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:09.006181 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.006698 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.007756 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.008147 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.008276 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.008339 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.008577 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.008646 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 00:34:09.009438 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b968d76b434735afc51e525afc8893" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.009622 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.005455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:35187 (local address 127.10.11.193:34325)
0927 00:34:09.005569 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.005574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.006094 (+ 520us) server_negotiation.cc:407] Connection header received
0927 00:34:09.006263 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.006271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.006322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.006403 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:09.007007 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.007516 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.008581 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.009099 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.009217 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.009250 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.009445 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.009498 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:34:09.009641 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.005403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34325 (local address 127.10.11.194:35187)
0927 00:34:09.006004 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.006016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.006154 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.006416 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.006418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.006434 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:09.006881 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.006888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.007635 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.007640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.008456 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.008465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.009233 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.009237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.009327 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.009348 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.009428 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.009498 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":328,"thread_start_us":76,"threads_started":1}
I20260927 00:34:09.009721 18242 raft_consensus.cc:2454] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.010207 18035 leader_election.cc:302] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 6cc765f76eb24ce1b975ef89a9660e0d; no voters:
I20260927 00:34:09.010237 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b968d76b434735afc51e525afc8893" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" is_pre_election: true
I20260927 00:34:09.010452 18322 raft_consensus.cc:2781] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:09.010501 17960 raft_consensus.cc:2454] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.010548 18322 raft_consensus.cc:482] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:09.010592 18322 raft_consensus.cc:3037] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.011581 18322 raft_consensus.cc:504] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:09.011888 18322 leader_election.cc:288] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857), 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325)
I20260927 00:34:09.012208 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b968d76b434735afc51e525afc8893" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
I20260927 00:34:09.012228 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b968d76b434735afc51e525afc8893" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
I20260927 00:34:09.012333 18242 raft_consensus.cc:3037] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.012354 17960 raft_consensus.cc:3037] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.013329 18242 raft_consensus.cc:2454] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.013413 17960 raft_consensus.cc:2454] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.013621 18035 leader_election.cc:302] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 6cc765f76eb24ce1b975ef89a9660e0d; no voters:
I20260927 00:34:09.013810 18322 raft_consensus.cc:2781] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:09.013996 18322 raft_consensus.cc:686] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Running, Role: LEADER
I20260927 00:34:09.014326 18322 consensus_queue.cc:237] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } }
I20260927 00:34:09.017467 17804 catalog_manager.cc:5495] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "1ccc69c039a046959ecf8e5de45941b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } health_report { overall_health: UNKNOWN } } }
I20260927 00:34:09.043859 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.040311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.10.11.194:33071)
0927 00:34:09.040407 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.040411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.040425 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:09.040612 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.040615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.040664 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.040746 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:09.040974 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.041456 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.042449 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.042650 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.042724 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.042981 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.043380 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.043462 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.043609 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.043689 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:34:09.049261 17960 raft_consensus.cc:1261] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.049675 18242 raft_consensus.cc:1261] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.049955 18326 consensus_queue.cc:1035] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.050248 18322 consensus_queue.cc:1035] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.064405 17804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.064555 17804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.066751 17804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Sending DeleteTablet for 3 replicas of tablet d0b968d76b434735afc51e525afc8893
I20260927 00:34:09.067381 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b968d76b434735afc51e525afc8893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:44618
I20260927 00:34:09.067464 18082 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b968d76b434735afc51e525afc8893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:49810
I20260927 00:34:09.067838 18339 tablet_replica.cc:330] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: stopping tablet replica
I20260927 00:34:09.067873 17942 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b968d76b434735afc51e525afc8893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260927 00:34:09.068133 18339 raft_consensus.cc:2229] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.068303 18341 tablet_replica.cc:330] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: stopping tablet replica
I20260927 00:34:09.068421 18339 raft_consensus.cc:2258] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.068782 18341 raft_consensus.cc:2229] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.068926 17804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:34:09.069271 18341 raft_consensus.cc:2258] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:34:09.069300 17804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:09.069833 18339 ts_tablet_manager.cc:1905] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.070686 18340 tablet_replica.cc:330] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: stopping tablet replica
I20260927 00:34:09.070698 18341 ts_tablet_manager.cc:1905] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.072052 18340 raft_consensus.cc:2229] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:09.072355 18340 raft_consensus.cc:2258] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.073151 18339 ts_tablet_manager.cc:1918] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.073249 18339 log.cc:1196] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/d0b968d76b434735afc51e525afc8893
I20260927 00:34:09.073578 18339 ts_tablet_manager.cc:1939] T d0b968d76b434735afc51e525afc8893 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting consensus metadata
I20260927 00:34:09.073912 18082 tablet_service.cc:1467] Processing CreateTablet for tablet c20e3f03a26942d6a55858d31d9ac629 (DEFAULT_TABLE table=test-table [id=938db3476de7452f84669bfa6f8022b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.074084 18340 ts_tablet_manager.cc:1905] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.074180 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20e3f03a26942d6a55858d31d9ac629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.074316 17942 tablet_service.cc:1467] Processing CreateTablet for tablet c20e3f03a26942d6a55858d31d9ac629 (DEFAULT_TABLE table=test-table [id=938db3476de7452f84669bfa6f8022b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.074668 18214 tablet_service.cc:1467] Processing CreateTablet for tablet c20e3f03a26942d6a55858d31d9ac629 (DEFAULT_TABLE table=test-table [id=938db3476de7452f84669bfa6f8022b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.074870 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20e3f03a26942d6a55858d31d9ac629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.076073 18340 ts_tablet_manager.cc:1918] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.076133 18340 log.cc:1196] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/d0b968d76b434735afc51e525afc8893
I20260927 00:34:09.076327 18340 ts_tablet_manager.cc:1939] T d0b968d76b434735afc51e525afc8893 P 1ccc69c039a046959ecf8e5de45941b4: Deleting consensus metadata
I20260927 00:34:09.076647 18316 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Bootstrap starting.
I20260927 00:34:09.076948 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20e3f03a26942d6a55858d31d9ac629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.077147 17791 catalog_manager.cc:4841] TS 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857): tablet d0b968d76b434735afc51e525afc8893 (table test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]) successfully deleted
I20260927 00:34:09.077525 18316 tablet_bootstrap.cc:654] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.078025 18318 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Bootstrap starting.
I20260927 00:34:09.078567 18316 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: No bootstrap required, opened a new log
I20260927 00:34:09.078634 18316 ts_tablet_manager.cc:1397] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:34:09.078931 18318 tablet_bootstrap.cc:654] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.079118 18316 raft_consensus.cc:348] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.079218 18316 raft_consensus.cc:374] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.079257 18316 raft_consensus.cc:729] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.079375 18316 consensus_queue.cc:260] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.079509 17790 catalog_manager.cc:4841] TS 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071): tablet d0b968d76b434735afc51e525afc8893 (table test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]) successfully deleted
I20260927 00:34:09.079643 18316 ts_tablet_manager.cc:1428] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.079852 18317 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Bootstrap starting.
I20260927 00:34:09.080128 18341 ts_tablet_manager.cc:1918] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.080183 18318 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: No bootstrap required, opened a new log
I20260927 00:34:09.080276 18318 ts_tablet_manager.cc:1397] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:34:09.080286 18341 log.cc:1196] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/d0b968d76b434735afc51e525afc8893
I20260927 00:34:09.080655 18341 ts_tablet_manager.cc:1939] T d0b968d76b434735afc51e525afc8893 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting consensus metadata
I20260927 00:34:09.080721 18317 tablet_bootstrap.cc:654] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.080711 18318 raft_consensus.cc:348] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.080821 18318 raft_consensus.cc:374] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.080863 18318 raft_consensus.cc:729] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc765f76eb24ce1b975ef89a9660e0d, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.081018 18318 consensus_queue.cc:260] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.081279 18318 ts_tablet_manager.cc:1428] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.081969 17792 catalog_manager.cc:4841] TS 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325): tablet d0b968d76b434735afc51e525afc8893 (table test-table [id=d17e7cf0f90f49c2baaa2f92ea8252c0]) successfully deleted
I20260927 00:34:09.082917 18317 tablet_bootstrap.cc:492] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: No bootstrap required, opened a new log
I20260927 00:34:09.083004 18317 ts_tablet_manager.cc:1397] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:34:09.083743 18317 raft_consensus.cc:348] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.083864 18317 raft_consensus.cc:374] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.083904 18317 raft_consensus.cc:729] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.084028 18317 consensus_queue.cc:260] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.084273 18317 ts_tablet_manager.cc:1428] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.102352 18324 raft_consensus.cc:482] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.102451 18324 raft_consensus.cc:504] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.103494 18324 leader_election.cc:288] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.103968 18334 raft_consensus.cc:482] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.104043 18334 raft_consensus.cc:504] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.104341 18334 leader_election.cc:288] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.104707 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.104813 18242 raft_consensus.cc:2454] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.105082 18035 leader_election.cc:302] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 6cc765f76eb24ce1b975ef89a9660e0d; no voters:
I20260927 00:34:09.105568 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" is_pre_election: true
I20260927 00:34:09.105672 17960 raft_consensus.cc:2454] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.105963 18334 raft_consensus.cc:2781] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:09.106017 18334 raft_consensus.cc:482] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:09.106055 18334 raft_consensus.cc:3037] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.107163 18334 raft_consensus.cc:504] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.107448 18334 leader_election.cc:288] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Requested vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.107949 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
I20260927 00:34:09.108047 17960 raft_consensus.cc:3037] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.109014 17960 raft_consensus.cc:2454] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.109211 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
I20260927 00:34:09.109308 18242 raft_consensus.cc:3037] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.109544 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.103776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:42469 (local address 127.10.11.194:33071)
0927 00:34:09.103920 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.103924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.104560 (+ 636us) server_negotiation.cc:407] Connection header received
0927 00:34:09.104746 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.104750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.104800 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.104867 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:09.105277 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.106043 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.108097 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.108708 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.108817 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.108876 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.109292 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.109372 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:34:09.110008 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.103745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:33071 (local address 127.10.11.193:42469)
0927 00:34:09.104478 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.104494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.104628 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.105006 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.105009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.105025 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:09.105137 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.105142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.106199 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.106203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.107950 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.107960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.109001 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.109019 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.109129 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.109155 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.109727 (+ 572us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.109820 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":690,"thread_start_us":104,"threads_started":1}
I20260927 00:34:09.110287 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.103963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:38789 (local address 127.10.11.195:42857)
0927 00:34:09.104079 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.104085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.104169 (+ 84us) server_negotiation.cc:407] Connection header received
0927 00:34:09.104407 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.104410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.104459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.104518 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:09.104980 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.105850 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.107811 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.108461 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.108549 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.108599 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.110032 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.110144 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:34:09.110291 18036 leader_election.cc:302] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.110473 18242 raft_consensus.cc:2454] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.110739 18102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccc69c039a046959ecf8e5de45941b4" is_pre_election: true
I20260927 00:34:09.110884 18334 raft_consensus.cc:2781] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:09.110920 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.103745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42857 (local address 127.10.11.193:38789)
0927 00:34:09.104078 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.104094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.104249 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.104673 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.104677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.104696 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:09.104842 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.104854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.105988 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.105991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.107668 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.107679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.109731 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.109738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.109862 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.109884 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.110634 (+ 750us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.110708 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":234,"thread_start_us":80,"threads_started":1}
I20260927 00:34:09.110960 18334 raft_consensus.cc:686] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Running, Role: LEADER
I20260927 00:34:09.111071 18334 consensus_queue.cc:237] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.111318 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20e3f03a26942d6a55858d31d9ac629" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.111485 18242 raft_consensus.cc:2379] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86cd2713f5e34c6da1c4dc1ff8a02301 in current term 1: Already voted for candidate 1ccc69c039a046959ecf8e5de45941b4 in this term.
I20260927 00:34:09.112324 17894 leader_election.cc:302] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86cd2713f5e34c6da1c4dc1ff8a02301; no voters: 1ccc69c039a046959ecf8e5de45941b4, 6cc765f76eb24ce1b975ef89a9660e0d
I20260927 00:34:09.112704 18324 raft_consensus.cc:2726] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:34:09.112834 17803 catalog_manager.cc:5495] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "1ccc69c039a046959ecf8e5de45941b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } health_report { overall_health: UNKNOWN } } }
W20260927 00:34:09.136601 18289 tablet.cc:2369] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469248, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.254@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:09.152787 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.139903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38255 (local address 127.0.0.1:33566)
0927 00:34:09.140166 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.140181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.140281 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.140521 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.140524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.140998 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:09.141100 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.141106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.141793 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.141796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.142711 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.142720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.143126 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.143130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.143215 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.147589 (+ 4374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:09.147604 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:09.149165 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:09.151073 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:09.151084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:09.151090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:09.151327 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:09.151621 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:09.151626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:09.151628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:09.151713 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:09.152155 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:09.152162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:09.152270 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.152393 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.152575 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":111,"threads_started":1}
I20260927 00:34:09.152786 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.139945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.10.11.254:38255)
0927 00:34:09.140072 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.140076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.140254 (+ 178us) server_negotiation.cc:407] Connection header received
0927 00:34:09.140371 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.140373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.140408 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.140474 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:09.141203 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.141694 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.142815 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.143024 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.143101 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.149313 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:09.149317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:09.149320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:09.149346 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:09.150951 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:09.151442 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:09.151447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:09.151449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:09.151527 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:09.151812 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:09.151818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:09.151821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:09.152013 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:09.152131 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.152439 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.152645 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":52}
I20260927 00:34:09.157023 17803 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:33566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:34:09.157714 17803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:09.162145 17942 tablet_service.cc:1467] Processing CreateTablet for tablet c0b103b151fa475099b0c8105d0da2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c3bc1bfc674055b7b2cced7055a82d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:34:09.162220 18082 tablet_service.cc:1467] Processing CreateTablet for tablet c0b103b151fa475099b0c8105d0da2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c3bc1bfc674055b7b2cced7055a82d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:34:09.162513 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b103b151fa475099b0c8105d0da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.162585 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b103b151fa475099b0c8105d0da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.164651 18316 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: Bootstrap starting.
I20260927 00:34:09.165283 18317 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: Bootstrap starting.
I20260927 00:34:09.165565 18316 tablet_bootstrap.cc:654] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.166028 18214 tablet_service.cc:1467] Processing CreateTablet for tablet c0b103b151fa475099b0c8105d0da2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c3bc1bfc674055b7b2cced7055a82d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:34:09.166343 18317 tablet_bootstrap.cc:654] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.166383 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b103b151fa475099b0c8105d0da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.167517 18317 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: No bootstrap required, opened a new log
I20260927 00:34:09.167517 18316 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: No bootstrap required, opened a new log
I20260927 00:34:09.167618 18316 ts_tablet_manager.cc:1397] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:34:09.167618 18317 ts_tablet_manager.cc:1397] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:09.168022 18317 raft_consensus.cc:348] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.168018 18316 raft_consensus.cc:348] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.168119 18317 raft_consensus.cc:374] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.168119 18316 raft_consensus.cc:374] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.168152 18317 raft_consensus.cc:729] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.168152 18316 raft_consensus.cc:729] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.168268 18316 consensus_queue.cc:260] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.168268 18317 consensus_queue.cc:260] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.168521 18316 ts_tablet_manager.cc:1428] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:34:09.168521 18317 ts_tablet_manager.cc:1428] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.169081 18334 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.169199 18334 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.169356 18324 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.169480 18324 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.170799 18318 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: Bootstrap starting.
I20260927 00:34:09.171761 18318 tablet_bootstrap.cc:654] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.177045 18318 tablet_bootstrap.cc:492] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: No bootstrap required, opened a new log
I20260927 00:34:09.177129 18318 ts_tablet_manager.cc:1397] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:34:09.177577 18318 raft_consensus.cc:348] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.177721 18318 raft_consensus.cc:374] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.177764 18318 raft_consensus.cc:729] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc765f76eb24ce1b975ef89a9660e0d, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.177898 18318 consensus_queue.cc:260] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.178210 18318 ts_tablet_manager.cc:1428] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.178408 18323 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.178529 18323 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 00:34:09.180097 18149 tablet.cc:2369] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:09.250233 18324 raft_consensus.cc:482] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.250362 18324 raft_consensus.cc:504] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.250838 18324 leader_election.cc:288] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.251255 18102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b103b151fa475099b0c8105d0da2d0" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccc69c039a046959ecf8e5de45941b4" is_pre_election: true
I20260927 00:34:09.251273 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b103b151fa475099b0c8105d0da2d0" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.251428 18242 raft_consensus.cc:2454] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86cd2713f5e34c6da1c4dc1ff8a02301 in term 0.
I20260927 00:34:09.251447 18102 raft_consensus.cc:2454] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86cd2713f5e34c6da1c4dc1ff8a02301 in term 0.
I20260927 00:34:09.251761 17895 leader_election.cc:302] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc765f76eb24ce1b975ef89a9660e0d, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.252038 18324 raft_consensus.cc:2781] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:09.252085 18324 raft_consensus.cc:482] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:09.252111 18324 raft_consensus.cc:3037] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.253073 18324 raft_consensus.cc:504] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.253396 18324 leader_election.cc:288] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 election: Requested vote from peers 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.253687 18102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b103b151fa475099b0c8105d0da2d0" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccc69c039a046959ecf8e5de45941b4"
I20260927 00:34:09.253810 18102 raft_consensus.cc:3037] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.253841 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b103b151fa475099b0c8105d0da2d0" candidate_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
I20260927 00:34:09.253964 18242 raft_consensus.cc:3037] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.254737 18102 raft_consensus.cc:2454] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86cd2713f5e34c6da1c4dc1ff8a02301 in term 1.
I20260927 00:34:09.255023 17894 leader_election.cc:302] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.255033 18242 raft_consensus.cc:2454] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86cd2713f5e34c6da1c4dc1ff8a02301 in term 1.
I20260927 00:34:09.255175 18324 raft_consensus.cc:2781] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:09.255429 18324 raft_consensus.cc:686] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 LEADER]: Becoming Leader. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Running, Role: LEADER
I20260927 00:34:09.255684 18324 consensus_queue.cc:237] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.256816 18357 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: TxnStatusTablet state changed. Reason: New leader 86cd2713f5e34c6da1c4dc1ff8a02301. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.256934 18357 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:34:09.257196 18359 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:34:09.257354 18359 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:34:09.257931 17805 catalog_manager.cc:5495] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } health_report { overall_health: UNKNOWN } } }
I20260927 00:34:09.278813 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.274744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34325 (local address 127.0.0.1:45684)
0927 00:34:09.274885 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.274902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.275009 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.275334 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.275336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.275342 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:34:09.275424 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.275428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.276426 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.276431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.277076 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.277085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.277684 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.277686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.277756 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.277799 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.278560 (+ 761us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.278573 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.278645 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.278712 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:34:09.278823 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.274856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.10.11.193:34325)
0927 00:34:09.274981 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.274985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.274999 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:09.275153 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.275156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.275215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.275319 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:09.275522 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.276289 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.277221 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.277566 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.277668 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.277937 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.278462 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.278569 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.278656 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.278705 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:34:09.281399 18242 raft_consensus.cc:1261] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Refusing update from remote peer 86cd2713f5e34c6da1c4dc1ff8a02301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.281539 18102 raft_consensus.cc:1261] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Refusing update from remote peer 86cd2713f5e34c6da1c4dc1ff8a02301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.281914 18357 consensus_queue.cc:1035] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.282066 18324 consensus_queue.cc:1035] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.283665 18334 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: TxnStatusTablet state changed. Reason: New leader 86cd2713f5e34c6da1c4dc1ff8a02301. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.283797 18334 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.283865 18323 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: TxnStatusTablet state changed. Reason: New leader 86cd2713f5e34c6da1c4dc1ff8a02301. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.283967 18323 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.285853 18357 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.285969 18357 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:34:09.286734 18324 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.286827 18324 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 86cd2713f5e34c6da1c4dc1ff8a02301: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:34:09.287066 18331 mvcc.cc:205] Tried to move back new op lower bound from 7333762045052502016 to 7333762044953636864. Current Snapshot: MvccSnapshot[applied={T|T < 7333762045052502016}]
I20260927 00:34:09.287237 18323 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.287322 18323 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 6cc765f76eb24ce1b975ef89a9660e0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.287645 18334 tablet_replica.cc:439] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } } }
I20260927 00:34:09.287747 18334 tablet_replica.cc:442] T c0b103b151fa475099b0c8105d0da2d0 P 1ccc69c039a046959ecf8e5de45941b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:34:09.302016 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.298345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34325 (local address 127.10.11.194:36041)
0927 00:34:09.298732 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.298745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.298860 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.299092 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.299094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.299100 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:34:09.299183 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.299187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.300169 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.300172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.300783 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.300800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.301368 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.301371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.301436 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.301474 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.301755 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.301760 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.301839 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.301909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":88,"threads_started":1}
I20260927 00:34:09.302032 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.298595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:36041 (local address 127.10.11.193:34325)
0927 00:34:09.298736 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.298740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.298791 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:34:09.298949 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.298952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.298997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.299074 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:09.299274 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.300040 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.300904 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.301256 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.301365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.301591 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.301661 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.301732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.301871 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.301916 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:34:09.307617 17960 raft_consensus.cc:1261] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.307616 18242 raft_consensus.cc:1261] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.307955 18334 consensus_queue.cc:1035] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.308072 18322 consensus_queue.cc:1035] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.310472 18329 mvcc.cc:205] Tried to move back new op lower bound from 7333762045160624128 to 7333762044360007680. Current Snapshot: MvccSnapshot[applied={T|T < 7333762045160624128}]
I20260927 00:34:09.313361 18330 mvcc.cc:205] Tried to move back new op lower bound from 7333762045160624128 to 7333762044360007680. Current Snapshot: MvccSnapshot[applied={T|T < 7333762045160624128}]
I20260927 00:34:09.328951 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.323915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:34089 (local address 127.10.11.194:33071)
0927 00:34:09.324024 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.324028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.324205 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:34:09.324337 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.324341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.324394 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.324474 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:09.324772 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.325550 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.326821 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.327047 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.327152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.327445 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.327544 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.327633 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.328747 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.328804 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:34:09.328975 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.323737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:33071 (local address 127.10.11.193:34089)
0927 00:34:09.324097 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.324112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:09.324221 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:09.324504 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:09.324507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:09.324515 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:34:09.324633 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.324639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.325695 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.325698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.326704 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:09.326715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.327171 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.327175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:09.327267 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.327315 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.328611 (+ 1296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.328619 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:34:09.328720 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:34:09.328796 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":137,"threads_started":1}
I20260927 00:34:09.350867 17804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.351025 17804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.352901 17804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Sending DeleteTablet for 3 replicas of tablet c20e3f03a26942d6a55858d31d9ac629
I20260927 00:34:09.353550 17942 tablet_service.cc:1514] Processing DeleteTablet for tablet c20e3f03a26942d6a55858d31d9ac629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260927 00:34:09.353569 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet c20e3f03a26942d6a55858d31d9ac629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:44618
I20260927 00:34:09.353714 18341 tablet_replica.cc:330] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: stopping tablet replica
I20260927 00:34:09.353714 18339 tablet_replica.cc:330] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: stopping tablet replica
I20260927 00:34:09.353812 18341 raft_consensus.cc:2229] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.353812 18339 raft_consensus.cc:2229] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.353920 18339 raft_consensus.cc:2258] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.353919 18341 raft_consensus.cc:2258] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.354074 18082 tablet_service.cc:1514] Processing DeleteTablet for tablet c20e3f03a26942d6a55858d31d9ac629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:49810
I20260927 00:34:09.354269 18340 tablet_replica.cc:330] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: stopping tablet replica
I20260927 00:34:09.354395 18340 raft_consensus.cc:2229] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:09.354593 18340 raft_consensus.cc:2258] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.354857 18339 ts_tablet_manager.cc:1905] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.355422 18340 ts_tablet_manager.cc:1905] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.357645 18339 ts_tablet_manager.cc:1918] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:34:09.357730 18339 log.cc:1196] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c20e3f03a26942d6a55858d31d9ac629
I20260927 00:34:09.357986 18339 ts_tablet_manager.cc:1939] T c20e3f03a26942d6a55858d31d9ac629 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting consensus metadata
I20260927 00:34:09.358039 18340 ts_tablet_manager.cc:1918] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:34:09.358140 18340 log.cc:1196] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c20e3f03a26942d6a55858d31d9ac629
I20260927 00:34:09.358392 18340 ts_tablet_manager.cc:1939] T c20e3f03a26942d6a55858d31d9ac629 P 1ccc69c039a046959ecf8e5de45941b4: Deleting consensus metadata
I20260927 00:34:09.358680 17791 catalog_manager.cc:4841] TS 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857): tablet c20e3f03a26942d6a55858d31d9ac629 (table test-table [id=938db3476de7452f84669bfa6f8022b5]) successfully deleted
I20260927 00:34:09.358935 18341 ts_tablet_manager.cc:1905] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.359043 17790 catalog_manager.cc:4841] TS 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071): tablet c20e3f03a26942d6a55858d31d9ac629 (table test-table [id=938db3476de7452f84669bfa6f8022b5]) successfully deleted
I20260927 00:34:09.360991 18341 ts_tablet_manager.cc:1918] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:34:09.361049 18341 log.cc:1196] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/c20e3f03a26942d6a55858d31d9ac629
I20260927 00:34:09.361274 18341 ts_tablet_manager.cc:1939] T c20e3f03a26942d6a55858d31d9ac629 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting consensus metadata
I20260927 00:34:09.361822 17792 catalog_manager.cc:4841] TS 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325): tablet c20e3f03a26942d6a55858d31d9ac629 (table test-table [id=938db3476de7452f84669bfa6f8022b5]) successfully deleted
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:09.385031 17854 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:09.378781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.10.11.254:38255)
0927 00:34:09.378902 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.378905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.379200 (+ 295us) server_negotiation.cc:407] Connection header received
0927 00:34:09.379355 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.379359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.379409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.379495 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:09.380164 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.380675 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.381521 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.381753 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.381861 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.384909 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33580: BlockingRecv error: recv got EOF from 127.0.0.1:33580 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM
I20260927 00:34:09.394522 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.386328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.10.11.254:38255)
0927 00:34:09.386439 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.386444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.386593 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:34:09.386705 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.386708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.386746 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.386836 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:09.387388 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.387924 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.388686 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.388932 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.389038 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.391499 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:09.391502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:09.391505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:09.391535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:09.392916 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:09.393326 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:09.393330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:09.393331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:09.393439 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:09.393723 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:09.393728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:09.393729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:09.393894 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:09.394028 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.394250 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.394353 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:34:09.396957 17804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:09.397360 17804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:09.402231 17942 tablet_service.cc:1467] Processing CreateTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 (DEFAULT_TABLE table=test-table [id=411afd14ffc941a39bc14e0585508d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.402513 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d995ca30d0424b8cff6f41ac18a457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.402494 18214 tablet_service.cc:1467] Processing CreateTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 (DEFAULT_TABLE table=test-table [id=411afd14ffc941a39bc14e0585508d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.402732 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d995ca30d0424b8cff6f41ac18a457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.404911 18318 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Bootstrap starting.
I20260927 00:34:09.405083 18082 tablet_service.cc:1467] Processing CreateTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 (DEFAULT_TABLE table=test-table [id=411afd14ffc941a39bc14e0585508d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.405355 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d995ca30d0424b8cff6f41ac18a457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.405879 18318 tablet_bootstrap.cc:654] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.406287 18317 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Bootstrap starting.
I20260927 00:34:09.407017 18318 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: No bootstrap required, opened a new log
I20260927 00:34:09.407088 18318 ts_tablet_manager.cc:1397] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:09.407166 18317 tablet_bootstrap.cc:654] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.407548 18318 raft_consensus.cc:348] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.407660 18318 raft_consensus.cc:374] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.407694 18318 raft_consensus.cc:729] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc765f76eb24ce1b975ef89a9660e0d, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.407776 18316 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Bootstrap starting.
I20260927 00:34:09.407819 18318 consensus_queue.cc:260] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.408087 18318 ts_tablet_manager.cc:1428] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.408581 18316 tablet_bootstrap.cc:654] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.409461 18317 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: No bootstrap required, opened a new log
I20260927 00:34:09.409548 18317 ts_tablet_manager.cc:1397] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:34:09.410044 18317 raft_consensus.cc:348] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.410213 18317 raft_consensus.cc:374] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.410264 18317 raft_consensus.cc:729] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.410398 18317 consensus_queue.cc:260] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.410708 18317 ts_tablet_manager.cc:1428] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.411079 18316 tablet_bootstrap.cc:492] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: No bootstrap required, opened a new log
I20260927 00:34:09.411166 18316 ts_tablet_manager.cc:1397] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 00:34:09.411556 18316 raft_consensus.cc:348] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.411664 18316 raft_consensus.cc:374] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.411705 18316 raft_consensus.cc:729] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.411826 18316 consensus_queue.cc:260] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.412093 18316 ts_tablet_manager.cc:1428] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.483812 18322 raft_consensus.cc:482] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.483934 18322 raft_consensus.cc:504] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.484419 18322 leader_election.cc:288] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.484750 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d995ca30d0424b8cff6f41ac18a457" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" is_pre_election: true
I20260927 00:34:09.484884 17960 raft_consensus.cc:2454] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.484869 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d995ca30d0424b8cff6f41ac18a457" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.485026 18242 raft_consensus.cc:2454] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.485173 18036 leader_election.cc:302] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.485347 18322 raft_consensus.cc:2781] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:09.485422 18322 raft_consensus.cc:482] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:09.485481 18322 raft_consensus.cc:3037] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.486720 18322 raft_consensus.cc:504] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.487118 18322 leader_election.cc:288] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Requested vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.487448 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d995ca30d0424b8cff6f41ac18a457" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
I20260927 00:34:09.487468 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d995ca30d0424b8cff6f41ac18a457" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
I20260927 00:34:09.487592 17960 raft_consensus.cc:3037] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.487592 18242 raft_consensus.cc:3037] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.488690 17960 raft_consensus.cc:2454] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.488690 18242 raft_consensus.cc:2454] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.488991 18035 leader_election.cc:302] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 6cc765f76eb24ce1b975ef89a9660e0d; no voters:
I20260927 00:34:09.489228 18322 raft_consensus.cc:2781] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:09.489349 18322 raft_consensus.cc:686] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Running, Role: LEADER
I20260927 00:34:09.489500 18322 consensus_queue.cc:237] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.491144 17803 catalog_manager.cc:5495] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "1ccc69c039a046959ecf8e5de45941b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } health_report { overall_health: UNKNOWN } } }
I20260927 00:34:09.508985 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.505625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.10.11.194:33071)
0927 00:34:09.505749 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:09.505754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:09.505837 (+ 83us) server_negotiation.cc:407] Connection header received
0927 00:34:09.506012 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:09.506015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:09.506077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:09.506203 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:09.506467 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.506972 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.507892 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:09.508120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:09.508246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:09.508519 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:09.508595 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:09.508678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:09.508801 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 00:34:09.508852 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:34:09.510975 17960 raft_consensus.cc:1261] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.511053 18242 raft_consensus.cc:1261] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.511397 18322 consensus_queue.cc:1035] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.511551 18334 consensus_queue.cc:1035] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.517403 17804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.517531 17804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.519641 17804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Sending DeleteTablet for 3 replicas of tablet c0d995ca30d0424b8cff6f41ac18a457
I20260927 00:34:09.520187 17942 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260927 00:34:09.520305 18082 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:49810
I20260927 00:34:09.520339 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d995ca30d0424b8cff6f41ac18a457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:44618
I20260927 00:34:09.520376 18341 tablet_replica.cc:330] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: stopping tablet replica
I20260927 00:34:09.520468 18340 tablet_replica.cc:330] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: stopping tablet replica
I20260927 00:34:09.520491 18341 raft_consensus.cc:2229] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.520577 18340 raft_consensus.cc:2229] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:09.520648 18341 raft_consensus.cc:2258] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.520736 18340 raft_consensus.cc:2258] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.520838 18339 tablet_replica.cc:330] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: stopping tablet replica
I20260927 00:34:09.520975 18339 raft_consensus.cc:2229] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.521109 18339 raft_consensus.cc:2258] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.521323 18340 ts_tablet_manager.cc:1905] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.521782 18341 ts_tablet_manager.cc:1905] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.522495 18339 ts_tablet_manager.cc:1905] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.523557 18340 ts_tablet_manager.cc:1918] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.523646 18340 log.cc:1196] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c0d995ca30d0424b8cff6f41ac18a457
I20260927 00:34:09.523844 18341 ts_tablet_manager.cc:1918] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.523928 18340 ts_tablet_manager.cc:1939] T c0d995ca30d0424b8cff6f41ac18a457 P 1ccc69c039a046959ecf8e5de45941b4: Deleting consensus metadata
I20260927 00:34:09.523939 18341 log.cc:1196] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/c0d995ca30d0424b8cff6f41ac18a457
I20260927 00:34:09.524185 18341 ts_tablet_manager.cc:1939] T c0d995ca30d0424b8cff6f41ac18a457 P 86cd2713f5e34c6da1c4dc1ff8a02301: Deleting consensus metadata
I20260927 00:34:09.524521 17790 catalog_manager.cc:4841] TS 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071): tablet c0d995ca30d0424b8cff6f41ac18a457 (table test-table [id=411afd14ffc941a39bc14e0585508d9e]) successfully deleted
I20260927 00:34:09.524863 18339 ts_tablet_manager.cc:1918] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:09.524943 18339 log.cc:1196] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c0d995ca30d0424b8cff6f41ac18a457
I20260927 00:34:09.524902 17792 catalog_manager.cc:4841] TS 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325): tablet c0d995ca30d0424b8cff6f41ac18a457 (table test-table [id=411afd14ffc941a39bc14e0585508d9e]) successfully deleted
I20260927 00:34:09.525134 18339 ts_tablet_manager.cc:1939] T c0d995ca30d0424b8cff6f41ac18a457 P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting consensus metadata
I20260927 00:34:09.525830 17791 catalog_manager.cc:4841] TS 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857): tablet c0d995ca30d0424b8cff6f41ac18a457 (table test-table [id=411afd14ffc941a39bc14e0585508d9e]) successfully deleted
I20260927 00:34:09.526027 17804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:09.526436 17804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:09.530598 17942 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe (DEFAULT_TABLE table=test-table [id=883adb9c053b410fb9e0033535747f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.530845 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0928d8b5d4032a22f37f90e32acbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.530794 18082 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe (DEFAULT_TABLE table=test-table [id=883adb9c053b410fb9e0033535747f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.531026 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0928d8b5d4032a22f37f90e32acbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.532644 18317 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: Bootstrap starting.
I20260927 00:34:09.532835 18214 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe (DEFAULT_TABLE table=test-table [id=883adb9c053b410fb9e0033535747f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:09.533048 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0928d8b5d4032a22f37f90e32acbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:09.533465 18317 tablet_bootstrap.cc:654] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.534948 18318 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Bootstrap starting.
I20260927 00:34:09.534970 18316 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Bootstrap starting.
I20260927 00:34:09.535588 18316 tablet_bootstrap.cc:654] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.535763 18318 tablet_bootstrap.cc:654] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:09.536685 18316 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: No bootstrap required, opened a new log
I20260927 00:34:09.536672 18317 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: No bootstrap required, opened a new log
I20260927 00:34:09.536825 18317 ts_tablet_manager.cc:1397] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 00:34:09.536827 18316 ts_tablet_manager.cc:1397] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:34:09.537186 18316 raft_consensus.cc:348] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.537204 18317 raft_consensus.cc:348] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.537287 18317 raft_consensus.cc:374] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.537308 18317 raft_consensus.cc:729] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cd2713f5e34c6da1c4dc1ff8a02301, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.537314 18316 raft_consensus.cc:374] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.537367 18316 raft_consensus.cc:729] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.537442 18317 consensus_queue.cc:260] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.537499 18316 consensus_queue.cc:260] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.537700 18317 ts_tablet_manager.cc:1428] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.537809 18316 ts_tablet_manager.cc:1428] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:34:09.538944 18318 tablet_bootstrap.cc:492] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: No bootstrap required, opened a new log
I20260927 00:34:09.539043 18318 ts_tablet_manager.cc:1397] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:34:09.539521 18318 raft_consensus.cc:348] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.539628 18318 raft_consensus.cc:374] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:09.539664 18318 raft_consensus.cc:729] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc765f76eb24ce1b975ef89a9660e0d, State: Initialized, Role: FOLLOWER
I20260927 00:34:09.539793 18318 consensus_queue.cc:260] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.540077 18318 ts_tablet_manager.cc:1428] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:34:09.570973 18322 raft_consensus.cc:482] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:09.571065 18322 raft_consensus.cc:504] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.571480 18322 leader_election.cc:288] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.571864 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0928d8b5d4032a22f37f90e32acbe" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" is_pre_election: true
I20260927 00:34:09.571890 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0928d8b5d4032a22f37f90e32acbe" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" is_pre_election: true
I20260927 00:34:09.572013 17960 raft_consensus.cc:2454] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.572023 18242 raft_consensus.cc:2454] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 0.
I20260927 00:34:09.572327 18036 leader_election.cc:302] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.572489 18322 raft_consensus.cc:2781] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:09.572548 18322 raft_consensus.cc:482] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:09.572592 18322 raft_consensus.cc:3037] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.573436 18322 raft_consensus.cc:504] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.573740 18322 leader_election.cc:288] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Requested vote from peers 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325), 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857)
I20260927 00:34:09.574033 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0928d8b5d4032a22f37f90e32acbe" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301"
I20260927 00:34:09.574142 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0928d8b5d4032a22f37f90e32acbe" candidate_uuid: "1ccc69c039a046959ecf8e5de45941b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc765f76eb24ce1b975ef89a9660e0d"
I20260927 00:34:09.574195 17960 raft_consensus.cc:3037] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.574254 18242 raft_consensus.cc:3037] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:09.575016 18242 raft_consensus.cc:2454] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.575019 17960 raft_consensus.cc:2454] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccc69c039a046959ecf8e5de45941b4 in term 1.
I20260927 00:34:09.575320 18036 leader_election.cc:302] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccc69c039a046959ecf8e5de45941b4, 86cd2713f5e34c6da1c4dc1ff8a02301; no voters:
I20260927 00:34:09.575558 18322 raft_consensus.cc:2781] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:09.575692 18322 raft_consensus.cc:686] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1ccc69c039a046959ecf8e5de45941b4, State: Running, Role: LEADER
I20260927 00:34:09.575834 18322 consensus_queue.cc:237] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } }
I20260927 00:34:09.577414 17803 catalog_manager.cc:5495] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "1ccc69c039a046959ecf8e5de45941b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccc69c039a046959ecf8e5de45941b4" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 } health_report { overall_health: UNKNOWN } } }
I20260927 00:34:09.592746 17960 raft_consensus.cc:1261] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.592759 18241 raft_consensus.cc:1261] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Refusing update from remote peer 1ccc69c039a046959ecf8e5de45941b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:09.593133 18322 consensus_queue.cc:1035] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc765f76eb24ce1b975ef89a9660e0d" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.593480 18322 consensus_queue.cc:1035] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cd2713f5e34c6da1c4dc1ff8a02301" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:09.623147 17805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.623277 17805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:09.625058 17805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beab0ed945a24b8a80db413e4eeaa9a6: Sending DeleteTablet for 3 replicas of tablet 38a0928d8b5d4032a22f37f90e32acbe
I20260927 00:34:09.625540 18082 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:49810
I20260927 00:34:09.625710 18340 tablet_replica.cc:330] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: stopping tablet replica
I20260927 00:34:09.625684 17942 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260927 00:34:09.625754 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0928d8b5d4032a22f37f90e32acbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:09 UTC) from {username='oryx', principal='oryx/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:44618
I20260927 00:34:09.625828 18341 tablet_replica.cc:330] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301: stopping tablet replica
I20260927 00:34:09.625861 18340 raft_consensus.cc:2229] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:09.625918 18341 raft_consensus.cc:2229] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.625929 18339 tablet_replica.cc:330] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: stopping tablet replica
I20260927 00:34:09.626001 18341 raft_consensus.cc:2258] T 38a0928d8b5d4032a22f37f90e32acbe P 86cd2713f5e34c6da1c4dc1ff8a02301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.626012 18340 raft_consensus.cc:2258] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.626052 18339 raft_consensus.cc:2229] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:09.626178 18339 raft_consensus.cc:2258] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:09.626562 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 17877
I20260927 00:34:09.626653 18340 ts_tablet_manager.cc:1905] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.626850 18339 ts_tablet_manager.cc:1905] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:09.629103 18340 ts_tablet_manager.cc:1918] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:34:09.629103 18339 ts_tablet_manager.cc:1918] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:34:09.629169 18339 log.cc:1196] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/38a0928d8b5d4032a22f37f90e32acbe
I20260927 00:34:09.629169 18340 log.cc:1196] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/38a0928d8b5d4032a22f37f90e32acbe
I20260927 00:34:09.629427 18340 ts_tablet_manager.cc:1939] T 38a0928d8b5d4032a22f37f90e32acbe P 1ccc69c039a046959ecf8e5de45941b4: Deleting consensus metadata
I20260927 00:34:09.629427 18339 ts_tablet_manager.cc:1939] T 38a0928d8b5d4032a22f37f90e32acbe P 6cc765f76eb24ce1b975ef89a9660e0d: Deleting consensus metadata
I20260927 00:34:09.630007 17791 catalog_manager.cc:4841] TS 6cc765f76eb24ce1b975ef89a9660e0d (127.10.11.195:42857): tablet 38a0928d8b5d4032a22f37f90e32acbe (table test-table [id=883adb9c053b410fb9e0033535747f2d]) successfully deleted
I20260927 00:34:09.630146 17790 catalog_manager.cc:4841] TS 1ccc69c039a046959ecf8e5de45941b4 (127.10.11.194:33071): tablet 38a0928d8b5d4032a22f37f90e32acbe (table test-table [id=883adb9c053b410fb9e0033535747f2d]) successfully deleted
W20260927 00:34:09.635449 17792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:34325 (error 108)
I20260927 00:34:09.636086 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18017
I20260927 00:34:09.636991 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:09.636082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34325 (local address 127.0.0.1:45686)
0927 00:34:09.636748 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:09.636870 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":65,"threads_started":1}
W20260927 00:34:09.638036 17792 catalog_manager.cc:4584] TS 86cd2713f5e34c6da1c4dc1ff8a02301 (127.10.11.193:34325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38a0928d8b5d4032a22f37f90e32acbe: Network error: Client connection negotiation failed: client connection to 127.10.11.193:34325: connect: Connection refused (error 111)
I20260927 00:34:09.643252 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18157
I20260927 00:34:09.648765 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 17773
2026-09-27T00:34:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2760 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[18403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[18403](info): set up 2 sockets
Sep 27 00:34:09 dist-test-slave-6gxc krb5kdc[18403](info): commencing operation
krb5kdc: starting...
W20260927 00:34:10.675091 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:10 dist-test-slave-6gxc krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:34:10.786963 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34147
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:10.889755 18416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:10.889986 18416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:10.890023 18416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:10.893850 18416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:10.893903 18416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:10.893922 18416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:10.893939 18416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:10.893996 18416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:10.899012 18416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:34147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:10.899891 18416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:10.900803 18416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:10.908998 18421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:10.909158 18416 server_base.cc:1034] running on GCE node
W20260927 00:34:10.909018 18424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:10.909080 18422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:10.909740 18416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:10.910822 18416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:10.912029 18416 hybrid_clock.cc:648] HybridClock initialized: now 1790469250912001 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:34:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1263 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:10 dist-test-slave-6gxc krb5kdc[18429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:10 dist-test-slave-6gxc krb5kdc[18429](info): set up 2 sockets
Sep 27 00:34:10 dist-test-slave-6gxc krb5kdc[18429](info): commencing operation
krb5kdc: starting...
W20260927 00:34:11.951234 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:34:12.062201 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39259
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:12.165267 18442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:12.165486 18442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:12.165529 18442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:12.169433 18442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:12.169480 18442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:12.169500 18442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:12.169516 18442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:12.169531 18442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:12.174621 18442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:12.175482 18442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:12.176366 18442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:12.184202 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.184208 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.184208 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:12.184216 18442 server_base.cc:1034] running on GCE node
I20260927 00:34:12.184909 18442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:12.185844 18442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:12.186998 18442 hybrid_clock.cc:648] HybridClock initialized: now 1790469252186975 us; error 37 us; skew 500 ppm
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:12.189458 18442 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:34:12.190343 18442 webserver.cc:460] Webserver started at http://127.10.11.254:37579/ using document root <none> and password file <none>
I20260927 00:34:12.190840 18442 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:12.190896 18442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:12.191109 18442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:12.192912 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "879165ba627d47eb82007349c2ee7982"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "f459392020cc1bf7c05963d8019063a2"
server_key_iv: "e0b70834d0357e8b3fb2e5203da2ae4c"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.193416 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "879165ba627d47eb82007349c2ee7982"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "f459392020cc1bf7c05963d8019063a2"
server_key_iv: "e0b70834d0357e8b3fb2e5203da2ae4c"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.196802 18442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:34:12.198976 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.200069 18442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:12.200183 18442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "879165ba627d47eb82007349c2ee7982"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "f459392020cc1bf7c05963d8019063a2"
server_key_iv: "e0b70834d0357e8b3fb2e5203da2ae4c"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.200274 18442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:12.224700 18442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:12.227613 18442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:12.227795 18442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:12.234302 18442 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:39259
I20260927 00:34:12.234336 18509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:39259 every 8 connection(s)
I20260927 00:34:12.235122 18442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:12.236802 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18442
I20260927 00:34:12.236937 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:12.237217 10287 external_mini_cluster.cc:1428] Setting key de73130a0ae631ddea7349f22bba4988
I20260927 00:34:12.237849 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:12.243084 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Bootstrap starting.
I20260927 00:34:12.245124 18510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Neither blocks nor log segments found. Creating new log.
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:12.245950 18510 log.cc:824] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:12.247828 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: No bootstrap required, opened a new log
I20260927 00:34:12.250476 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.238535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.10.11.254:39259)
0927 00:34:12.238933 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:12.238945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:12.238975 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:34:12.239551 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:12.239566 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:12.239837 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:12.240153 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:12.240911 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.241847 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.242744 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.243203 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.243361 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.246136 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:12.246142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:12.246164 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:12.246190 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:12.247782 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.248248 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.248252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.248256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.248364 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.248642 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.248644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.248645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.248998 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:12.249133 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:12.249393 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 00:34:12.249660 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":142,"threads_started":1}
I20260927 00:34:12.251906 18510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:12.252151 18510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:12.252225 18510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879165ba627d47eb82007349c2ee7982, State: Initialized, Role: FOLLOWER
I20260927 00:34:12.252651 18510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:12.252764 18510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:12.252841 18510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:12.252933 18510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:12.253690 18510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:12.254000 18510 leader_election.cc:302] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879165ba627d47eb82007349c2ee7982; no voters:
I20260927 00:34:12.254302 18510 leader_election.cc:288] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:12.254391 18515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:12.254640 18515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 LEADER]: Becoming Leader. State: Replica: 879165ba627d47eb82007349c2ee7982, State: Running, Role: LEADER
I20260927 00:34:12.254907 18515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:12.255373 18510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:12.256412 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "879165ba627d47eb82007349c2ee7982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } } }
I20260927 00:34:12.256546 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:12.256815 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879165ba627d47eb82007349c2ee7982. Latest consensus state: current_term: 1 leader_uuid: "879165ba627d47eb82007349c2ee7982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } } }
I20260927 00:34:12.256888 18524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:12.256913 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:12.259877 18524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:12.265051 18524 catalog_manager.cc:1347] Generated new cluster ID: 98ef3f05f82c467ab896e82ddf7aa287
I20260927 00:34:12.265120 18524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:12.280879 18524 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:12.281957 18524 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:12.290342 18524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Generated new TSK 0
I20260927 00:34:12.290998 18524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:34:12.320917 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:12.449071 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:12.449278 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:12.449316 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:12.453346 18536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:12.453393 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:12.453481 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:12.458820 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:12.459679 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:12.460600 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:12.469928 18542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.469941 18544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.469941 18541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:12.470211 18536 server_base.cc:1034] running on GCE node
I20260927 00:34:12.470604 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:12.471144 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:12.472344 18536 hybrid_clock.cc:648] HybridClock initialized: now 1790469252472335 us; error 25 us; skew 500 ppm
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:12.474923 18536 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:34:12.475670 18536 webserver.cc:460] Webserver started at http://127.10.11.193:42737/ using document root <none> and password file <none>
I20260927 00:34:12.476204 18536 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:12.476300 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:12.476509 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:12.478406 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "b02652b0b89746cc8f9718c2d24873e2"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "a2173f2547bb2721bd237dac5a1b2d14"
server_key_iv: "e2c66f126d5fada907f39b4609e070a5"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.478924 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "b02652b0b89746cc8f9718c2d24873e2"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "a2173f2547bb2721bd237dac5a1b2d14"
server_key_iv: "e2c66f126d5fada907f39b4609e070a5"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.482326 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:34:12.484437 18551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.485473 18536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:12.485606 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "b02652b0b89746cc8f9718c2d24873e2"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "a2173f2547bb2721bd237dac5a1b2d14"
server_key_iv: "e2c66f126d5fada907f39b4609e070a5"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.485707 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:12.495976 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:12.498581 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:12.498777 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:12.499293 18536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:12.500012 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:12.500083 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.500150 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:12.500202 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.508870 18536 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:46325
I20260927 00:34:12.508895 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:46325 every 8 connection(s)
I20260927 00:34:12.509627 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:12.516825 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18536
I20260927 00:34:12.516995 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:12.517319 10287 external_mini_cluster.cc:1428] Setting key 883d150f6d910d0b970957867031073e
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:12.523384 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.511256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:36053 (local address 127.10.11.254:39259)
0927 00:34:12.511400 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:12.511405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:12.512407 (+ 1002us) server_negotiation.cc:407] Connection header received
0927 00:34:12.513202 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:12.513205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:12.513249 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:12.513332 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:12.514468 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.515021 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.516357 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.516799 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.516911 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.519563 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:12.519570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:12.519577 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:12.519609 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:12.521157 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.521776 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.521778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.521780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.521859 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.522259 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.522262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.522263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.522444 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:12.522527 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:12.523164 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 00:34:12.523271 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:34:12.524479 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.511642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:36053)
0927 00:34:12.512256 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:12.512301 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:12.513016 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:12.513489 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:12.513495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:12.513831 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:12.514339 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.514354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.515143 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.515146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.516225 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.516241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.517014 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.517020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.517218 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.517742 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:12.517767 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:12.519397 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:12.521274 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.521283 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.521300 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.521669 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.521949 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.521954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.521956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.522156 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.522556 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:12.522566 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:12.522863 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 00:34:12.523090 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 00:34:12.523417 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":108,"threads_started":1}
I20260927 00:34:12.525720 18665 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39259
I20260927 00:34:12.525952 18665 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:12.526525 18665 heartbeater.cc:511] Master 127.10.11.254:39259 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:34:12.528002 18474 ts_manager.cc:194] Registered new tserver with Master: b02652b0b89746cc8f9718c2d24873e2 (127.10.11.193:46325)
I20260927 00:34:12.529084 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:36053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:34:12.543375 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:12.646755 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:12.647001 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:12.647071 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:12.651053 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:12.651144 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:12.651214 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:12.656190 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:12.657073 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:12.657953 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:12.666707 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.666724 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:12.667008 18670 server_base.cc:1034] running on GCE node
W20260927 00:34:12.667075 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:12.667423 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:12.668041 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:12.669216 18670 hybrid_clock.cc:648] HybridClock initialized: now 1790469252669203 us; error 24 us; skew 500 ppm
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:12.671780 18670 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:34:12.672538 18670 webserver.cc:460] Webserver started at http://127.10.11.194:35479/ using document root <none> and password file <none>
I20260927 00:34:12.673041 18670 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:12.673128 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:12.673336 18670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:12.675217 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "c327091628cf497aa049b0347d100ff6"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "784e064d88071b159594f24fb9f027e1"
server_key_iv: "52512b3ee1ffaadc3da66b0ea88ea976"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.675750 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "c327091628cf497aa049b0347d100ff6"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "784e064d88071b159594f24fb9f027e1"
server_key_iv: "52512b3ee1ffaadc3da66b0ea88ea976"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.679236 18670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:12.681460 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.682610 18670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:12.682737 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "c327091628cf497aa049b0347d100ff6"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "784e064d88071b159594f24fb9f027e1"
server_key_iv: "52512b3ee1ffaadc3da66b0ea88ea976"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.682837 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:12.692538 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:12.695111 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:12.695312 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:12.695809 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:12.696528 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:12.696605 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.696694 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:12.696744 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.705101 18670 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:44849
I20260927 00:34:12.705164 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:44849 every 8 connection(s)
I20260927 00:34:12.705886 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:12.708407 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18670
I20260927 00:34:12.708539 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:12.708806 10287 external_mini_cluster.cc:1428] Setting key 52642c67a22d313fbfbed86593da0dcb
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:34:12.720186 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.707524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:39833 (local address 127.10.11.254:39259)
0927 00:34:12.707643 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:12.707652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:12.708527 (+ 875us) server_negotiation.cc:407] Connection header received
0927 00:34:12.709495 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:12.709498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:12.709552 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:12.709645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:12.710831 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.711665 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.713375 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.713786 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.713906 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.716591 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:12.716597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:12.716604 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:12.716635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:12.718280 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.718765 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.718768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.718769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.718839 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.719232 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.719235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.719237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.719408 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:12.719489 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:12.719961 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 00:34:12.720075 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:34:12.721068 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.707788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:39833)
0927 00:34:12.708384 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:12.708419 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:12.709236 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:12.709774 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:12.709780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:12.710252 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:12.710694 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.710706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.711807 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.711812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.713261 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.713270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.713956 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.713959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.714090 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.715077 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:12.715092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:12.716433 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:12.718430 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.718434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.718447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.718675 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.718940 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.718943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.718945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.719080 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.719504 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:12.719508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:12.719704 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 00:34:12.719850 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:34:12.720218 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":119,"threads_started":1}
I20260927 00:34:12.722093 18799 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39259
I20260927 00:34:12.722321 18799 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:12.722750 18799 heartbeater.cc:511] Master 127.10.11.254:39259 requested a full tablet report, sending...
I20260927 00:34:12.723695 18474 ts_manager.cc:194] Registered new tserver with Master: c327091628cf497aa049b0347d100ff6 (127.10.11.194:44849)
I20260927 00:34:12.724242 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:39833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:34:12.734779 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:12.840940 18804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:12.841164 18804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:12.841203 18804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:12.845095 18804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:12.845151 18804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:12.845206 18804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:12.850278 18804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:12.851282 18804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:12.852344 18804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:12.861606 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.861763 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:12.861605 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:12.862002 18804 server_base.cc:1034] running on GCE node
I20260927 00:34:12.862447 18804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:12.863101 18804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:12.864289 18804 hybrid_clock.cc:648] HybridClock initialized: now 1790469252864267 us; error 34 us; skew 500 ppm
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:12.867100 18804 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:34:12.867976 18804 webserver.cc:460] Webserver started at http://127.10.11.195:34983/ using document root <none> and password file <none>
I20260927 00:34:12.868516 18804 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:12.868606 18804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:12.868825 18804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:12.870903 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "1f1015f911a346cc9cd2643878e9ee06"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "ed877aa6aeb453799f7af4112d498c94"
server_key_iv: "47cccdf892ebe1b30b2f6694a160c1f4"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.871434 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "1f1015f911a346cc9cd2643878e9ee06"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "ed877aa6aeb453799f7af4112d498c94"
server_key_iv: "47cccdf892ebe1b30b2f6694a160c1f4"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.875005 18804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:12.877321 18819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.878528 18804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:12.878651 18804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "1f1015f911a346cc9cd2643878e9ee06"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "ed877aa6aeb453799f7af4112d498c94"
server_key_iv: "47cccdf892ebe1b30b2f6694a160c1f4"
server_key_version: "encryptionkey@0"
I20260927 00:34:12.878750 18804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:12.893224 18804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:12.895946 18804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:12.896124 18804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:12.896633 18804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:12.897372 18804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:12.897440 18804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.897507 18804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:12.897550 18804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:12.906288 18804 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:36419
I20260927 00:34:12.906337 18932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:36419 every 8 connection(s)
I20260927 00:34:12.907079 18804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:12.910408 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18804
I20260927 00:34:12.910531 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:12.910806 10287 external_mini_cluster.cc:1428] Setting key c7ad508c849e7953b550de3b0763a6be
Sep 27 00:34:12 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:12.920298 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.908775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:41043 (local address 127.10.11.254:39259)
0927 00:34:12.908930 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:12.908934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:12.909959 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 00:34:12.911160 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:12.911163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:12.911230 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:12.911322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:12.912735 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.913326 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.914472 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.914688 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.914786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.917066 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:12.917069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:12.917071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:12.917093 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:12.918546 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.919040 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.919042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.919043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.919115 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:12.919423 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:12.919426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:12.919427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:12.919589 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:12.919665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:12.920101 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 00:34:12.920189 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:34:12.921047 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:12.909138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:41043)
0927 00:34:12.909782 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:12.909835 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:12.910963 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:12.911490 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:12.911498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:12.911948 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:12.912563 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.912581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.913439 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.913442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.914366 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:12.914375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:12.914820 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:12.914823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:12.914950 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:12.915498 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:12.915513 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:12.916922 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:12.918655 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.918660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.918670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.918948 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.919196 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:12.919198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:12.919199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:12.919330 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:12.919677 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:12.919682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:12.919904 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:34:12.920057 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:34:12.920284 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":109,"threads_started":1}
I20260927 00:34:12.922235 18933 heartbeater.cc:347] Connected to a master server at 127.10.11.254:39259
I20260927 00:34:12.922444 18933 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:12.922845 18933 heartbeater.cc:511] Master 127.10.11.254:39259 requested a full tablet report, sending...
I20260927 00:34:12.923831 18474 ts_manager.cc:194] Registered new tserver with Master: 1f1015f911a346cc9cd2643878e9ee06 (127.10.11.195:36419)
I20260927 00:34:12.924403 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:41043
I20260927 00:34:12.924525 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
Entry for principal oryx/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab.
WARNING: no policy specified for oryx/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
Entry for principal oryx/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab.
WARNING: no policy specified for oryx/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
Entry for principal oryx/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab.
I20260927 00:34:12.992745 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18536
I20260927 00:34:12.998273 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18670
I20260927 00:34:13.003037 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18804
I20260927 00:34:13.008205 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18442
I20260927 00:34:13.013952 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39259
--webserver_interface=127.10.11.254
--webserver_port=37579
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:13.120112 18942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:13.120361 18942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:13.120402 18942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:13.124346 18942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:13.124398 18942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:13.124421 18942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:13.124467 18942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:13.124503 18942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:13.129462 18942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=37579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:13.130409 18942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:13.131325 18942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:13.139351 18947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:13.139361 18948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.139400 18942 server_base.cc:1034] running on GCE node
W20260927 00:34:13.139601 18950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.140144 18942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:13.141158 18942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:13.142349 18942 hybrid_clock.cc:648] HybridClock initialized: now 1790469253142332 us; error 34 us; skew 500 ppm
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469253, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:13.144970 18942 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:34:13.145824 18942 webserver.cc:460] Webserver started at http://127.10.11.254:37579/ using document root <none> and password file <none>
I20260927 00:34:13.146402 18942 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:13.146508 18942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:13.150045 18942 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 00:34:13.152081 18957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.153403 18942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:34:13.153493 18942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "879165ba627d47eb82007349c2ee7982"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "f459392020cc1bf7c05963d8019063a2"
server_key_iv: "e0b70834d0357e8b3fb2e5203da2ae4c"
server_key_version: "encryptionkey@0"
I20260927 00:34:13.154072 18942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:13.184878 18942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:13.187884 18942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:13.188028 18942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:13.194180 18942 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:39259
I20260927 00:34:13.194188 19009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:39259 every 8 connection(s)
I20260927 00:34:13.195132 18942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:13.199023 19010 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:34:13.199473 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 18942
I20260927 00:34:13.199878 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:46325
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:39259
--webserver_port=42737
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:34:13.200609 19010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Bootstrap starting.
I20260927 00:34:13.216595 19010 log.cc:824] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:13.222608 19010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:34:13.222949 19010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Bootstrap complete.
I20260927 00:34:13.226941 19010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:13.227300 19010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879165ba627d47eb82007349c2ee7982, State: Initialized, Role: FOLLOWER
I20260927 00:34:13.227769 19010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:13.227880 19010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:13.227965 19010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:13.228078 19010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:34:13.229806 19010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:13.230166 19010 leader_election.cc:302] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879165ba627d47eb82007349c2ee7982; no voters:
I20260927 00:34:13.230490 19010 leader_election.cc:288] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:34:13.230581 19014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:34:13.230791 19014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [term 2 LEADER]: Becoming Leader. State: Replica: 879165ba627d47eb82007349c2ee7982, State: Running, Role: LEADER
I20260927 00:34:13.231071 19014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } }
I20260927 00:34:13.231640 19010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:13.232587 19016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879165ba627d47eb82007349c2ee7982. Latest consensus state: current_term: 2 leader_uuid: "879165ba627d47eb82007349c2ee7982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } } }
I20260927 00:34:13.232715 19016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:13.232583 19015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "879165ba627d47eb82007349c2ee7982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879165ba627d47eb82007349c2ee7982" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39259 } } }
I20260927 00:34:13.233000 19015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982 [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:13.233649 19023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:13.236727 19023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:13.237322 19023 catalog_manager.cc:1259] Loaded cluster ID: 98ef3f05f82c467ab896e82ddf7aa287
I20260927 00:34:13.237380 19023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:13.239629 19023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:13.240223 19023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 879165ba627d47eb82007349c2ee7982: Loaded TSK: 0
I20260927 00:34:13.241334 19023 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:34:13.306676 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:13.306885 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:13.306924 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:13.310840 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:13.310885 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:13.310959 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:13.315897 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.193.keytab
--principal=oryx/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:46325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=42737
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:13.316753 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:13.317651 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:13.327373 19039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:13.327390 19036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.327653 19012 server_base.cc:1034] running on GCE node
W20260927 00:34:13.327680 19037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.328155 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:13.328716 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:13.329895 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790469253329868 us; error 49 us; skew 500 ppm
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469253, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:13.332764 19012 init.cc:377] Logged in from keytab as oryx/127.10.11.193@KRBTEST.COM (short username oryx)
I20260927 00:34:13.333716 19012 webserver.cc:460] Webserver started at http://127.10.11.193:42737/ using document root <none> and password file <none>
I20260927 00:34:13.334306 19012 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:13.334404 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:13.338634 19012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:13.340557 19046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.341748 19012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:13.341835 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "b02652b0b89746cc8f9718c2d24873e2"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "a2173f2547bb2721bd237dac5a1b2d14"
server_key_iv: "e2c66f126d5fada907f39b4609e070a5"
server_key_version: "encryptionkey@0"
I20260927 00:34:13.342401 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:13.355065 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:13.357864 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:13.358047 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:13.358692 19012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:13.359493 19012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:13.359540 19012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.359576 19012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:13.359627 19012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.370072 19012 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:46325
I20260927 00:34:13.370157 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:46325 every 8 connection(s)
I20260927 00:34:13.371012 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:13.375593 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19012
I20260927 00:34:13.376029 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:44849
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:39259
--webserver_port=35479
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:13.383433 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.372848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:38277 (local address 127.10.11.254:39259)
0927 00:34:13.373292 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.373317 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.373780 (+ 463us) server_negotiation.cc:407] Connection header received
0927 00:34:13.374884 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.374899 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.375161 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.375485 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.376645 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.377569 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.378737 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.379240 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.379417 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.382815 (+ 3398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:38277: BlockingRecv error: recv got EOF from 127.10.11.193:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260927 00:34:13.383598 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.372993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:38277)
0927 00:34:13.373632 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.373668 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.374514 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.375551 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.375558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.375966 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.376477 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.376492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.377703 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.377707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.378613 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.378621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.379404 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.379409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.379586 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.380130 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.380145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.382655 (+ 2510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":113,"threads_started":1}
W20260927 00:34:13.384016 19160 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:39259 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.11.254:39259: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:13.389384 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.384549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52687 (local address 127.10.11.254:39259)
0927 00:34:13.384683 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.384688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.384699 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:34:13.384751 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.384753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.384784 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.384868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.385215 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.385693 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.386504 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.386713 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.386825 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.389199 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:52687: BlockingRecv error: recv got EOF from 127.10.11.193:52687 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:34:13.389410 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.384437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:52687)
0927 00:34:13.384521 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.384535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.384621 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.384879 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.384882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.385032 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.385112 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.385117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.385806 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.385809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.386394 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.386402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.386823 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.386827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.386914 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.387354 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.387364 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.389220 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:13.394590 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.389835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:35295)
0927 00:34:13.389917 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.389931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.389994 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.390205 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.390207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.390317 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.390389 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.390393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.391351 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.391354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.391843 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.391850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.392396 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.392398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.392453 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.392866 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.392877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.394478 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 00:34:13.394773 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.389901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:35295 (local address 127.10.11.254:39259)
0927 00:34:13.390007 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.390011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.390023 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:13.390057 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.390060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.390095 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.390195 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.390502 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.391248 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.391953 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.392287 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.392384 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.394575 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:35295: BlockingRecv error: recv got EOF from 127.10.11.193:35295 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:34:13.394775 19160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:34:13.482651 19164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:13.482892 19164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:13.482967 19164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:13.486876 19164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:13.486945 19164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:13.487064 19164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:13.492110 19164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.194.keytab
--principal=oryx/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:44849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=35479
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:13.493045 19164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:13.493950 19164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:13.502696 19170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:13.502662 19169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.502851 19164 server_base.cc:1034] running on GCE node
W20260927 00:34:13.502692 19172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.503279 19164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:13.503867 19164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:13.505046 19164 hybrid_clock.cc:648] HybridClock initialized: now 1790469253505035 us; error 24 us; skew 500 ppm
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469253, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:13.507699 19164 init.cc:377] Logged in from keytab as oryx/127.10.11.194@KRBTEST.COM (short username oryx)
I20260927 00:34:13.508513 19164 webserver.cc:460] Webserver started at http://127.10.11.194:35479/ using document root <none> and password file <none>
I20260927 00:34:13.509023 19164 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:13.509121 19164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:13.512593 19164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:13.514568 19179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.515699 19164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:13.515822 19164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "c327091628cf497aa049b0347d100ff6"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "784e064d88071b159594f24fb9f027e1"
server_key_iv: "52512b3ee1ffaadc3da66b0ea88ea976"
server_key_version: "encryptionkey@0"
I20260927 00:34:13.516331 19164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:13.527513 19164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:13.530447 19164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:13.530620 19164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:13.531116 19164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:13.531877 19164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:13.531953 19164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.532013 19164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:13.532068 19164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.541222 19164 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:44849
I20260927 00:34:13.541247 19292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:44849 every 8 connection(s)
I20260927 00:34:13.542089 19164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:13.551450 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19164
I20260927 00:34:13.551891 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:36419
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:39259
--webserver_port=34983
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:13.552148 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.543854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:32825 (local address 127.10.11.254:39259)
0927 00:34:13.544054 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.544059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.544967 (+ 908us) server_negotiation.cc:407] Connection header received
0927 00:34:13.545845 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.545847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.545893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.545972 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.547168 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.547686 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.548734 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.548936 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.549019 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.552035 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:32825: BlockingRecv error: recv got EOF from 127.10.11.194:32825 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 00:34:13.552660 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.544209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:32825)
0927 00:34:13.544791 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.544827 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.545679 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.546190 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.546197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.546524 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.547042 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.547055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.547783 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.547785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.548637 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.548645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.549061 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.549064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.549193 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.549663 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.549678 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.551926 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":101,"threads_started":1}
W20260927 00:34:13.553157 19293 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:39259 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.11.254:39259: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:13.559508 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.553665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:51601)
0927 00:34:13.553821 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.553837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.553903 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.554152 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.554154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.554331 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.554449 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.554467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.555488 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.555491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.556088 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.556096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.556869 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.556872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.556943 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.557531 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.557543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.559419 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 00:34:13.559747 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.553747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:51601 (local address 127.10.11.254:39259)
0927 00:34:13.553855 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.553858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.553870 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:13.553986 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.553988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.554020 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.554089 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.554568 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.555378 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.556212 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.556646 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.556781 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.559598 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:51601: BlockingRecv error: recv got EOF from 127.10.11.194:51601 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:13.565212 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.560044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:56763)
0927 00:34:13.560142 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.560157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.560230 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.560508 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.560510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.560622 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.560680 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.560684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.561727 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.561728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.562273 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.562281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.562921 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.562924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.562983 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.563452 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.563462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.565136 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:34:13.565382 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.560148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:56763 (local address 127.10.11.254:39259)
0927 00:34:13.560266 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.560272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.560288 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:34:13.560327 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.560332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.560394 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.560491 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.560787 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.561624 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.562388 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.562784 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.562915 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.565273 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:56763: BlockingRecv error: recv got EOF from 127.10.11.194:56763 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 00:34:13.565445 19293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:34:13.659595 19296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:13.659801 19296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:13.659840 19296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:13.663635 19296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:13.663679 19296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:13.663733 19296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:13.668638 19296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/krb5kdc/oryx_127.10.11.195.keytab
--principal=oryx/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:36419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=34983
--tserver_master_addrs=127.10.11.254:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:13.669472 19296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:13.670418 19296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:13.679113 19304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:13.679169 19301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.679358 19296 server_base.cc:1034] running on GCE node
W20260927 00:34:13.679139 19302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:13.679855 19296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:13.680440 19296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:13.681594 19296 hybrid_clock.cc:648] HybridClock initialized: now 1790469253681580 us; error 26 us; skew 500 ppm
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469253, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:13.684235 19296 init.cc:377] Logged in from keytab as oryx/127.10.11.195@KRBTEST.COM (short username oryx)
I20260927 00:34:13.685024 19296 webserver.cc:460] Webserver started at http://127.10.11.195:34983/ using document root <none> and password file <none>
I20260927 00:34:13.685489 19296 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:13.685554 19296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:13.689066 19296 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:13.690949 19311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.692065 19296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:13.692175 19296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "1f1015f911a346cc9cd2643878e9ee06"
format_stamp: "Formatted at 2026-09-27 00:34:12 on dist-test-slave-6gxc"
server_key: "ed877aa6aeb453799f7af4112d498c94"
server_key_iv: "47cccdf892ebe1b30b2f6694a160c1f4"
server_key_version: "encryptionkey@0"
I20260927 00:34:13.692653 19296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:13.717577 19296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:13.720851 19296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:13.721047 19296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:13.721606 19296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:13.722440 19296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:13.722487 19296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.722527 19296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:13.722579 19296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:13.731117 19296 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:36419
I20260927 00:34:13.731144 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:36419 every 8 connection(s)
I20260927 00:34:13.732067 19296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:13.738556 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19296
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:13.743059 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.734021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:46789 (local address 127.10.11.254:39259)
0927 00:34:13.734189 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.734194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.735225 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 00:34:13.736030 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.736032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.736075 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.736146 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.737252 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.737745 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.739208 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.739545 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.739626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.742945 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:46789: BlockingRecv error: recv got EOF from 127.10.11.195:46789 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:34:13.743640 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.734441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:46789)
0927 00:34:13.735078 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.735115 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.735865 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.736347 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.736357 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.736693 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.737136 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.737148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.737871 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.737874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.739065 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.739078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.739820 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.739822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.739964 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.740474 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.740489 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.742850 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":101,"threads_started":1}
W20260927 00:34:13.744205 19425 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:39259 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.11.254:39259: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
I20260927 00:34:13.748701 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.739580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.10.11.254:39259)
0927 00:34:13.739813 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.739817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.739828 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:34:13.739865 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.739867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.739897 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.740008 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.740820 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.741572 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.742657 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.742900 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.743024 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.744537 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:13.744544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:13.744560 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:13.744596 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:13.746559 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:13.747022 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:13.747028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:13.747034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:13.747188 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:13.747450 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:13.747452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:13.747453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:13.747873 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:13.748060 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:13.748289 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:34:13.748513 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:13.750602 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.744655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:57459 (local address 127.10.11.254:39259)
0927 00:34:13.744759 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.744761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.744773 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:13.744909 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.744913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.744956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.745040 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.745503 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.746086 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.747174 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.747527 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.747599 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.750495 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:57459: BlockingRecv error: recv got EOF from 127.10.11.195:57459 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:34:13.750497 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.744606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:57459)
0927 00:34:13.744703 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.744718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.744793 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.745071 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.745073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.745233 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.745347 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.745358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.746224 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.746228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.747061 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.747072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.747632 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.747636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.747726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.748505 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.748518 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.750366 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:13 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:13.755637 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:13.751230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:43443)
0927 00:34:13.751366 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:13.751380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:13.751447 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:13.751663 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:13.751666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:13.751813 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:13.751871 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.751876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.752547 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.752552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.753067 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:13.753074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.753594 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.753598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:13.753674 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.754016 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:13.754026 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:13.755516 (+ 1490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 00:34:13.755729 19162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:13.751306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:43443 (local address 127.10.11.254:39259)
0927 00:34:13.751420 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:13.751425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:13.751437 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:13.751528 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:13.751531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:13.751563 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:13.751651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:13.751976 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.752438 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.753177 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:13.753482 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:13.753594 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:13.755610 (+ 2016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:43443: BlockingRecv error: recv got EOF from 127.10.11.195:43443 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:34:13.755884 19425 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:14.405493 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:14.395452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:44745)
0927 00:34:14.395743 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:14.395761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:14.395865 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:14.396125 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:14.396128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:14.396349 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:14.396589 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.396598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.397344 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.397350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.397926 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.397933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.398449 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.398453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.398555 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.403553 (+ 4998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:14.403564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:14.405377 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260927 00:34:14.405642 19431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:14.395555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44745 (local address 127.10.11.254:39259)
0927 00:34:14.395842 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:14.395865 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:14.395882 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:34:14.395946 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:14.395950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:14.396000 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:14.396107 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:14.396707 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.397241 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.398052 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.398338 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.398452 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.405510 (+ 7058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:44745: BlockingRecv error: recv got EOF from 127.10.11.193:44745 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:14.571557 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:14.566086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:57599)
0927 00:34:14.566414 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:14.566438 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:14.566572 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:14.566851 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:14.566853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:14.567027 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:14.567141 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.567147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.567854 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.567857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.568485 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.568493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.568905 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.568909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.568991 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.569567 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:14.569589 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:14.571453 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
W20260927 00:34:14.571730 19431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:14.566184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:57599 (local address 127.10.11.254:39259)
0927 00:34:14.566333 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:14.566339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:14.566534 (+ 195us) server_negotiation.cc:407] Connection header received
0927 00:34:14.566699 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:14.566701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:14.566743 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:14.566808 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:14.567239 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.567751 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.568582 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.568785 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.568856 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.571597 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:57599: BlockingRecv error: recv got EOF from 127.10.11.194:57599 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:14 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:14.761979 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:14.756405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:55553)
0927 00:34:14.756656 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:14.756670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:14.756768 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:14.757018 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:14.757021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:14.757239 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:14.757377 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.757387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.758417 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.758420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.758996 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:14.759004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.759385 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.759388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:14.759447 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.760013 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:14.760027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:14.761799 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
W20260927 00:34:14.762008 19431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:14.756461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:55553 (local address 127.10.11.254:39259)
0927 00:34:14.756584 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:14.756587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:14.756733 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:34:14.756845 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:14.756847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:14.756904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:14.756974 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:14.757537 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.758325 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.759088 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:14.759294 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:14.759369 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:14.761906 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:55553: BlockingRecv error: recv got EOF from 127.10.11.195:55553 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:15.411403 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:15.406227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:37819)
0927 00:34:15.406498 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:15.406511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:15.406598 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:15.406821 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:15.406823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:15.407002 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:15.407099 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.407104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.407796 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.407798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.408357 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.408364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.408784 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.408787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.408847 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.409360 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:15.409371 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:15.411294 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
W20260927 00:34:15.411556 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:15.406303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:37819 (local address 127.10.11.254:39259)
0927 00:34:15.406560 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:15.406563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:15.406575 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:15.406679 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:15.406681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:15.406721 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:15.406794 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:15.407178 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.407704 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.408465 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.408684 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.408764 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.411430 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:37819: BlockingRecv error: recv got EOF from 127.10.11.193:37819 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:15.577901 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:15.572391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:59839)
0927 00:34:15.572682 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:15.572694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:15.572783 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:15.573050 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:15.573052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:15.573273 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:15.573434 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.573442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.574396 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.574399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.574948 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.574956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.575345 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.575348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.575416 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.575962 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:15.575976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:15.577741 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
W20260927 00:34:15.577955 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:15.572469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:59839 (local address 127.10.11.254:39259)
0927 00:34:15.572616 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:15.572619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:15.572768 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:34:15.572856 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:15.572860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:15.572908 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:15.573024 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:15.573556 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.574296 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.575050 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.575253 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.575326 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.577852 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:59839: BlockingRecv error: recv got EOF from 127.10.11.194:59839 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:15 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:15.768229 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:15.762750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:56341)
0927 00:34:15.763123 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:15.763135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:15.763259 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:15.763539 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:15.763542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:15.763735 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:15.763821 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.763827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.764643 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.764646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.765199 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:15.765206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.765659 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.765663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:15.765749 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.766300 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:15.766312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:15.768102 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":162,"threads_started":1}
W20260927 00:34:15.768354 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:15.762915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:56341 (local address 127.10.11.254:39259)
0927 00:34:15.763073 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:15.763077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:15.763202 (+ 125us) server_negotiation.cc:407] Connection header received
0927 00:34:15.763373 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:15.763376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:15.763418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:15.763519 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:15.763928 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.764528 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.765316 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:15.765551 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:15.765659 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:15.768242 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:56341: BlockingRecv error: recv got EOF from 127.10.11.195:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:16.418285 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:16.412252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:38715)
0927 00:34:16.412563 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:16.412576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:16.412667 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:16.412950 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:16.412952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:16.413129 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:16.413222 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.413227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.414135 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.414139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.415054 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.415066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.415524 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.415526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.415587 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.416114 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:16.416126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:16.418091 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
W20260927 00:34:16.418395 19439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:16.412360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:38715 (local address 127.10.11.254:39259)
0927 00:34:16.412621 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:16.412625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:16.412637 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:34:16.412756 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:16.412758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:16.412798 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:16.412893 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:16.413357 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.413977 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.415188 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.415422 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.415504 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.418248 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:38715: BlockingRecv error: recv got EOF from 127.10.11.193:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":55,"threads_started":1}
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:16.584451 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:16.578652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:54849)
0927 00:34:16.578935 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:16.578956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:16.579088 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:16.579367 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:16.579372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:16.579568 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:16.579658 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.579664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.580457 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.580461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.581039 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.581047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.581509 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.581513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.581618 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.582087 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:16.582146 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:16.584319 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
W20260927 00:34:16.584589 19439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:16.578717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54849 (local address 127.10.11.254:39259)
0927 00:34:16.578904 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:16.578908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:16.579030 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:34:16.579186 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:16.579190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:16.579237 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:16.579331 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:16.579781 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.580340 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.581157 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.581395 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.581507 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.584447 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:54849: BlockingRecv error: recv got EOF from 127.10.11.194:54849 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:16 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:16.774643 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:16.769035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:34283)
0927 00:34:16.769378 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:16.769393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:16.769528 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:16.769805 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:16.769808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:16.770025 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:16.770135 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.770140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.770928 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.770932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.771484 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:16.771491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.771950 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.771953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:16.772042 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.772578 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:16.772591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:16.774520 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
W20260927 00:34:16.774782 19439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:16.769172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34283 (local address 127.10.11.254:39259)
0927 00:34:16.769343 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:16.769348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:16.769479 (+ 131us) server_negotiation.cc:407] Connection header received
0927 00:34:16.769636 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:16.769640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:16.769695 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:16.769782 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:16.770254 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.770783 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.771605 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:16.771839 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:16.771949 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:16.774660 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:34283: BlockingRecv error: recv got EOF from 127.10.11.195:34283 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:17.424242 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:17.419060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:51595)
0927 00:34:17.419332 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:17.419349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:17.419453 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:17.419659 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:17.419661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:17.419825 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:17.419942 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.419947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.420700 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.420702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.421274 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.421282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.421687 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.421689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.421757 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.422337 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:17.422349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:17.424134 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
W20260927 00:34:17.424422 19443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:17.419128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:51595 (local address 127.10.11.254:39259)
0927 00:34:17.419459 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:17.419462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:17.419477 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:34:17.419510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:17.419512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:17.419551 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:17.419623 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:17.420036 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.420613 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.421374 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.421593 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.421669 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.424255 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:51595: BlockingRecv error: recv got EOF from 127.10.11.193:51595 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:17.590642 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:17.585161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:57687)
0927 00:34:17.585476 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:17.585489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:17.585619 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:17.585911 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:17.585914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:17.586180 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:17.586284 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.586290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.587035 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.587040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.587617 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.587625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.588065 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.588069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.588153 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.588652 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:17.588663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:17.590537 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
W20260927 00:34:17.590782 19443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:17.585198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:57687 (local address 127.10.11.254:39259)
0927 00:34:17.585368 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:17.585373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:17.585586 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:34:17.585726 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:17.585729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:17.585791 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:17.585884 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:17.586401 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.586924 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.587734 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.587960 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.588065 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.590669 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:57687: BlockingRecv error: recv got EOF from 127.10.11.194:57687 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:17 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:17.780956 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:17.775413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:48699)
0927 00:34:17.775711 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:17.775725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:17.775923 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:17.776215 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:17.776218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:17.776418 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:17.776506 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.776511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.777298 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.777300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.777849 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:17.777857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.778405 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.778409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:17.778501 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.779009 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:17.779021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:17.780848 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
W20260927 00:34:17.781103 19443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:17.775513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:48699 (local address 127.10.11.254:39259)
0927 00:34:17.775669 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:17.775673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:17.775808 (+ 135us) server_negotiation.cc:407] Connection header received
0927 00:34:17.776051 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:17.776056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:17.776106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:17.776195 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:17.776621 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.777172 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.777971 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:17.778288 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:17.778402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:17.780987 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:48699: BlockingRecv error: recv got EOF from 127.10.11.195:48699 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:18.430270 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:18.424960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:57453)
0927 00:34:18.425241 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:18.425255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:18.425337 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:18.425576 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:18.425578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:18.425749 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:18.425850 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.425855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.426578 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.426580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.427162 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.427170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.427578 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.427580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.427637 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.428095 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:18.428106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:18.430152 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
W20260927 00:34:18.430452 19447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:18.425067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:57453 (local address 127.10.11.254:39259)
0927 00:34:18.425296 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:18.425299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:18.425316 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:34:18.425422 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:18.425424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:18.425464 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:18.425542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:18.425943 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.426487 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.427258 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.427475 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.427559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.430305 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:57453: BlockingRecv error: recv got EOF from 127.10.11.193:57453 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":56,"threads_started":1}
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:18.597074 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:18.591466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:45087)
0927 00:34:18.591796 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:18.591812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:18.591922 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:18.592199 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:18.592202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:18.592457 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:18.592567 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.592574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.593485 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.593490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.594066 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.594074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.594581 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.594584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.594668 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.595125 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:18.595136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:18.596909 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
W20260927 00:34:18.597131 19447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:18.591626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:45087 (local address 127.10.11.254:39259)
0927 00:34:18.591834 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:18.591841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:18.591863 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:34:18.592027 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:18.592031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:18.592079 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:18.592180 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:18.592687 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.593393 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.594224 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.594469 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.594579 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.597026 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:45087: BlockingRecv error: recv got EOF from 127.10.11.194:45087 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:18 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:18.787171 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:18.781748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:37345)
0927 00:34:18.782013 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:18.782026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:18.782135 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:18.782374 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:18.782376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:18.782559 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:18.782648 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.782652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.783462 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.783465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.784027 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:18.784035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.784500 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.784503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:18.784561 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.785100 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:18.785112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:18.787064 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
W20260927 00:34:18.787295 19447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:18.781872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:37345 (local address 127.10.11.254:39259)
0927 00:34:18.782023 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:18.782026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:18.782074 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:34:18.782223 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:18.782226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:18.782271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:18.782340 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:18.782799 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.783353 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.784161 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:18.784393 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:18.784481 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:18.787182 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:37345: BlockingRecv error: recv got EOF from 127.10.11.195:37345 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:19.441377 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:19.431094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:44689)
0927 00:34:19.431442 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:19.431456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:19.431554 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:19.431812 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:19.431814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:19.431994 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:19.432087 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.432097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.432946 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.432949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.433575 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.433585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.434088 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.434091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.434216 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.439354 (+ 5138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:19.439367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:19.441264 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":102,"threads_started":1}
W20260927 00:34:19.441582 19451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:19.431216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44689 (local address 127.10.11.254:39259)
0927 00:34:19.431457 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:19.431460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:19.431507 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:34:19.431632 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:19.431634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:19.431674 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:19.431742 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:19.432255 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.432828 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.433689 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.433930 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.434065 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.441429 (+ 7364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:44689: BlockingRecv error: recv got EOF from 127.10.11.193:44689 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":57,"threads_started":1}
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:19.603024 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:19.597871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:55227)
0927 00:34:19.598156 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:19.598169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:19.598261 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:19.598478 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:19.598480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:19.598652 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:19.598742 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.598747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.599515 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.599517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.600071 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.600078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.600488 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.600491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.600549 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.601036 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:19.601047 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:19.602916 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
W20260927 00:34:19.603199 19451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:19.597906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:55227 (local address 127.10.11.254:39259)
0927 00:34:19.598053 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:19.598058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:19.598237 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:34:19.598331 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:19.598333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:19.598377 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:19.598443 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:19.598868 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.599421 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.600171 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.600397 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.600472 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.603050 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:55227: BlockingRecv error: recv got EOF from 127.10.11.194:55227 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:19 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:19.792986 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:19.788015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:52903)
0927 00:34:19.788261 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:19.788275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:19.788368 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:19.788578 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:19.788580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:19.788759 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:19.788849 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.788853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.789580 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.789582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.790210 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:19.790221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.790651 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.790653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:19.790710 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.791174 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:19.791185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:19.792864 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
W20260927 00:34:19.793107 19451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:19.788092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:52903 (local address 127.10.11.254:39259)
0927 00:34:19.788230 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:19.788234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:19.788324 (+ 90us) server_negotiation.cc:407] Connection header received
0927 00:34:19.788432 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:19.788434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:19.788476 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:19.788539 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:19.788952 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.789486 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.790324 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:19.790557 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:19.790637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:19.792996 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:52903: BlockingRecv error: recv got EOF from 127.10.11.195:52903 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:20.448230 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:20.442200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:54367)
0927 00:34:20.442509 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:20.442523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:20.442608 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:20.443012 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:20.443014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:20.443188 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:20.443279 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.443284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.444004 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.444008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.444865 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.444875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.445431 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.445434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.445542 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.446171 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:20.446185 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:20.448121 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
W20260927 00:34:20.448416 19455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:20.442616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:54367 (local address 127.10.11.254:39259)
0927 00:34:20.442818 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:20.442821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:20.442832 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:34:20.442863 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:20.442865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:20.442902 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:20.442977 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:20.443375 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.443892 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.444994 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.445224 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.445319 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.448243 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:54367: BlockingRecv error: recv got EOF from 127.10.11.193:54367 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:20.608959 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:20.603753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:39769)
0927 00:34:20.604087 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:20.604100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:20.604198 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:20.604412 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:20.604414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:20.604587 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:20.604676 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.604681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.605381 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.605384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.605951 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.605959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.606441 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.606444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.606511 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.607062 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:20.607073 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:20.608855 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
W20260927 00:34:20.609088 19455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:20.603789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:39769 (local address 127.10.11.254:39259)
0927 00:34:20.603928 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:20.603932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:20.604176 (+ 244us) server_negotiation.cc:407] Connection header received
0927 00:34:20.604263 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:20.604265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:20.604308 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:20.604371 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:20.604770 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.605284 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.606048 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.606318 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.606399 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.608976 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:39769: BlockingRecv error: recv got EOF from 127.10.11.194:39769 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:20 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:20.799468 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:20.793878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:50707)
0927 00:34:20.794201 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:20.794214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:20.794305 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:20.794535 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:20.794537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:20.794757 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:20.794869 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.794876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.795995 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.795998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.796614 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:20.796623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.797097 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.797100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:20.797158 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.797617 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:20.797631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:20.799364 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
W20260927 00:34:20.799667 19455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:20.793936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:50707 (local address 127.10.11.254:39259)
0927 00:34:20.794083 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:20.794090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:20.794282 (+ 192us) server_negotiation.cc:407] Connection header received
0927 00:34:20.794381 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:20.794384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:20.794428 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:20.794496 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:20.795005 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.795877 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.796708 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:20.796985 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:20.797074 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:20.799535 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:50707: BlockingRecv error: recv got EOF from 127.10.11.195:50707 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:21.454277 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:21.448933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:40503)
0927 00:34:21.449207 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:21.449220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:21.449305 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:21.449548 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:21.449549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:21.449717 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:21.449806 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.449810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.450530 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.450532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.451075 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.451082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.451527 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.451529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.451586 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.452057 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:21.452068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:21.454087 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
W20260927 00:34:21.454350 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:21.449076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:40503 (local address 127.10.11.254:39259)
0927 00:34:21.449274 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:21.449277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:21.449292 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:34:21.449394 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:21.449396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:21.449438 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:21.449515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:21.449901 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.450439 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.451180 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.451407 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.451487 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.454252 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:40503: BlockingRecv error: recv got EOF from 127.10.11.193:40503 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":56,"threads_started":1}
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:21.615602 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:21.609811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:44113)
0927 00:34:21.610089 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:21.610245 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:21.610369 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:21.610607 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:21.610609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:21.610781 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:21.610881 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.610886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.611761 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.611766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.612486 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.612498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.613096 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.613099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.613163 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.613662 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:21.613672 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:21.615489 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
W20260927 00:34:21.615782 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:21.609867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:44113 (local address 127.10.11.254:39259)
0927 00:34:21.610014 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:21.610018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:21.610326 (+ 308us) server_negotiation.cc:407] Connection header received
0927 00:34:21.610447 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:21.610449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:21.610498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:21.610567 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:21.610976 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.611647 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.612622 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.612994 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.613076 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.615624 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:44113: BlockingRecv error: recv got EOF from 127.10.11.194:44113 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:21 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:21.805516 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:21.800168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:54747)
0927 00:34:21.800483 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:21.800496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:21.800627 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:21.800888 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:21.800891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:21.801108 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:21.801197 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.801203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.801939 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.801944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.802569 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:21.802578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.803016 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.803019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:21.803103 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.803580 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:21.803592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:21.805356 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20260927 00:34:21.805604 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:21.800276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:54747 (local address 127.10.11.254:39259)
0927 00:34:21.800432 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:21.800436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:21.800565 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:34:21.800723 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:21.800728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:21.800779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:21.800866 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:21.801310 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.801833 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.802687 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:21.802907 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:21.803013 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:21.805484 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:54747: BlockingRecv error: recv got EOF from 127.10.11.195:54747 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:22.460749 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:22.455070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:51603)
0927 00:34:22.455431 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:22.455449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:22.455568 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:22.455833 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:22.455835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:22.456019 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:22.456114 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.456119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.456932 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.456937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.457573 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.457603 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.458054 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.458057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.458189 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.458762 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:22.458774 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:22.460639 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
W20260927 00:34:22.460891 19463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:22.455169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:51603 (local address 127.10.11.254:39259)
0927 00:34:22.455406 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:22.455410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:22.455532 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:34:22.455675 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:22.455678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:22.455720 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:22.455792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:22.456214 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.456738 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.457717 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.457939 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.458029 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.460760 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:51603: BlockingRecv error: recv got EOF from 127.10.11.193:51603 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":58,"threads_started":1}
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:22.621649 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:22.616312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:54777)
0927 00:34:22.616567 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:22.616580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:22.616676 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:22.616904 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:22.616907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:22.617082 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:22.617172 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.617177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.617866 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.617869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.618445 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.618453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.618874 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.618878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.618946 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.619471 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:22.619483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:22.621475 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
W20260927 00:34:22.621685 19463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:22.616434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54777 (local address 127.10.11.254:39259)
0927 00:34:22.616592 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:22.616595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:22.616630 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:34:22.616757 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:22.616759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:22.616801 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:22.616867 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:22.617266 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.617777 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.618543 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.618753 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.618832 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.621592 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:54777: BlockingRecv error: recv got EOF from 127.10.11.194:54777 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:22 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:22.811897 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:22.806314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:33575)
0927 00:34:22.806569 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:22.806582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:22.806678 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:22.806888 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:22.806890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:22.807060 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:22.807148 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.807153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.808025 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.808029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.808877 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:22.808886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.809347 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.809350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:22.809409 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.809868 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:22.809879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:22.811710 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
W20260927 00:34:22.811918 19463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:22.806408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:33575 (local address 127.10.11.254:39259)
0927 00:34:22.806542 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:22.806545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:22.806649 (+ 104us) server_negotiation.cc:407] Connection header received
0927 00:34:22.806742 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:22.806744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:22.806787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:22.806854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:22.807243 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.807910 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.809014 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:22.809245 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:22.809321 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:22.811821 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:33575: BlockingRecv error: recv got EOF from 127.10.11.195:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:23.466773 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:23.461451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:57547)
0927 00:34:23.461694 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:23.461712 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:23.461797 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:23.462027 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:23.462029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:23.462225 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:23.462317 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.462322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.463044 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.463047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.463615 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.463624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.464061 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.464064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.464129 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.464618 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:23.464629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:23.466599 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
W20260927 00:34:23.466889 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:23.461536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:57547 (local address 127.10.11.254:39259)
0927 00:34:23.461749 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:23.461752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:23.461765 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:23.461878 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:23.461880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:23.461919 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:23.462009 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:23.462409 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.462922 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.463719 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.463938 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.464022 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.466756 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:57547: BlockingRecv error: recv got EOF from 127.10.11.193:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:23.627861 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:23.622411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:45665)
0927 00:34:23.622779 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:23.622799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:23.622911 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:23.623188 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:23.623191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:23.623393 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:23.623478 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.623495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.624233 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.624238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.624802 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.624809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.625254 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.625258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.625341 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.625882 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:23.625894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:23.627699 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
W20260927 00:34:23.627912 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:23.622482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:45665 (local address 127.10.11.254:39259)
0927 00:34:23.622746 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:23.622751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:23.622868 (+ 117us) server_negotiation.cc:407] Connection header received
0927 00:34:23.623035 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:23.623038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:23.623078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:23.623164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:23.623602 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.624126 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.624915 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.625145 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.625252 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.627810 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:45665: BlockingRecv error: recv got EOF from 127.10.11.194:45665 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:23 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:23.817711 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:23.812714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:34985)
0927 00:34:23.812998 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:23.813011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:23.813107 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:23.813325 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:23.813327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:23.813500 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:23.813587 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.813592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.814347 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.814349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.814886 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:23.814893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.815300 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.815302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:23.815362 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.815817 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:23.815828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:23.817604 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260927 00:34:23.817831 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:23.812854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34985 (local address 127.10.11.254:39259)
0927 00:34:23.813011 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:23.813014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:23.813069 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:34:23.813187 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:23.813189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:23.813231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:23.813292 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:23.813686 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.814258 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.814995 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:23.815206 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:23.815284 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:23.817724 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:34985: BlockingRecv error: recv got EOF from 127.10.11.195:34985 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:24.476449 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:24.467581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:35193)
0927 00:34:24.467832 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:24.467854 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:24.467948 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:24.468176 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:24.468178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:24.468357 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:24.468451 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.468456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.469171 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.469173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.469722 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.469730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.470143 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.470146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.470204 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.474458 (+ 4254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:24.474470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:24.476264 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
W20260927 00:34:24.476532 19471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:24.467622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:35193 (local address 127.10.11.254:39259)
0927 00:34:24.467842 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:24.467845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:24.467920 (+ 75us) server_negotiation.cc:407] Connection header received
0927 00:34:24.468023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:24.468025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:24.468066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:24.468143 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:24.468547 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.469057 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.469817 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.470030 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.470153 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.476373 (+ 6220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:35193: BlockingRecv error: recv got EOF from 127.10.11.193:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:24.633898 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:24.628618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:57885)
0927 00:34:24.628870 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:24.628892 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:24.628987 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:24.629209 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:24.629211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:24.629397 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:24.629490 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.629495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.630228 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.630231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.630781 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.630790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.631309 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.631313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.631380 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.631938 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:24.631949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:24.633766 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
W20260927 00:34:24.633985 19471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:24.628669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:57885 (local address 127.10.11.254:39259)
0927 00:34:24.628804 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:24.628807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:24.628964 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:34:24.629062 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:24.629064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:24.629106 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:24.629172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:24.629601 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.630133 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.630879 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.631107 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.631200 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.633873 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:57885: BlockingRecv error: recv got EOF from 127.10.11.194:57885 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:24 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:24.823693 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:24.818654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:49923)
0927 00:34:24.818917 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:24.818929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:24.819019 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:24.819232 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:24.819234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:24.819400 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:24.819523 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.819529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.820255 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.820257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.820822 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:24.820829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.821252 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.821255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:24.821320 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.821779 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:24.821790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:24.823588 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260927 00:34:24.823900 19471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:24.818691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:49923 (local address 127.10.11.254:39259)
0927 00:34:24.818828 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:24.818831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:24.819000 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:34:24.819095 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:24.819097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:24.819142 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:24.819198 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:24.819603 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.820166 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.820913 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:24.821142 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:24.821217 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:24.823758 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:49923: BlockingRecv error: recv got EOF from 127.10.11.195:49923 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:25.482357 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:25.477191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:44441)
0927 00:34:25.477442 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:25.477454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:25.477538 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:25.477772 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:25.477774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:25.478040 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:25.478159 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.478165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.478863 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.478865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.479415 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.479421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.479811 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.479813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.479870 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.480330 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:25.480341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:25.482192 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
W20260927 00:34:25.482421 19475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:25.477247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44441 (local address 127.10.11.254:39259)
0927 00:34:25.477484 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:25.477487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:25.477500 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:25.477625 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:25.477627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:25.477668 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:25.477745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:25.478256 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.478776 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.479509 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.479723 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.479797 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.482301 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:44441: BlockingRecv error: recv got EOF from 127.10.11.193:44441 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":58,"threads_started":1}
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:25.639544 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:25.634575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:46673)
0927 00:34:25.634827 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:25.634839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:25.634931 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:25.635146 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:25.635148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:25.635311 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:25.635399 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.635403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.636112 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.636114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.636647 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.636654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.637065 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.637068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.637125 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.637616 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:25.637626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:25.639436 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
W20260927 00:34:25.639704 19475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:25.634624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:46673 (local address 127.10.11.254:39259)
0927 00:34:25.634778 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:25.634783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:25.634904 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:34:25.635013 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:25.635015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:25.635057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:25.635121 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:25.635492 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.636010 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.636748 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.636967 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.637046 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.639559 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:46673: BlockingRecv error: recv got EOF from 127.10.11.194:46673 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:25 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:25.830067 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:25.824604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:53499)
0927 00:34:25.824871 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:25.824885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:25.824984 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:25.825206 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:25.825208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:25.825383 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:25.825476 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.825481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.826402 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.826406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.827150 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:25.827161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.827640 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.827644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:25.827732 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.828205 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:25.828217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:25.829952 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260927 00:34:25.830227 19475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:25.824535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:53499 (local address 127.10.11.254:39259)
0927 00:34:25.824691 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:25.824694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:25.824954 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:34:25.825060 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:25.825062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:25.825105 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:25.825172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:25.825599 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.826281 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.827254 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:25.827499 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:25.827623 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:25.830075 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:53499: BlockingRecv error: recv got EOF from 127.10.11.195:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:26.488587 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:26.483131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:59849)
0927 00:34:26.483397 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:26.483410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:26.483498 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:26.483741 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:26.483743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:26.483938 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:26.484026 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.484031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.484779 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.484782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.485387 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.485395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.485839 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.485842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.485906 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.486523 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:26.486534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:26.488477 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
W20260927 00:34:26.488739 19479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:26.483221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:59849 (local address 127.10.11.254:39259)
0927 00:34:26.483467 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:26.483470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:26.483484 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:34:26.483579 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:26.483581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:26.483622 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:26.483712 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:26.484146 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.484667 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.485494 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.485710 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.485788 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.488607 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:59849: BlockingRecv error: recv got EOF from 127.10.11.193:59849 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:34:26.645720 19479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:26.640410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:59801 (local address 127.10.11.254:39259)
0927 00:34:26.640555 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:26.640562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:26.640651 (+ 89us) server_negotiation.cc:407] Connection header received
0927 00:34:26.640820 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:26.640823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:26.640873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:26.640958 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:26.641395 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.641917 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.642758 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.642985 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.643093 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.645600 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:59801: BlockingRecv error: recv got EOF from 127.10.11.194:59801 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:34:26.645824 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:26.640251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:59801)
0927 00:34:26.640552 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:26.640569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:26.640712 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:26.640975 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:26.640978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:26.641194 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:26.641278 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.641284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.642028 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.642032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.642620 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.642646 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.643093 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.643097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.643181 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.643664 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:26.643675 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:26.645478 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:26 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:26.836870 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:26.830921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:55943)
0927 00:34:26.831195 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:26.831208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:26.831384 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:26.831690 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:26.831693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:26.831936 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:26.832027 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.832033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.832920 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.832926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.833489 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:26.833497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.834015 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.834018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:26.834138 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.834643 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:26.834655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:26.836755 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
W20260927 00:34:26.837080 19479 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:26.830947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:55943 (local address 127.10.11.254:39259)
0927 00:34:26.831094 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:26.831098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:26.831314 (+ 216us) server_negotiation.cc:407] Connection header received
0927 00:34:26.831520 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:26.831523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:26.831569 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:26.831664 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:26.832144 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.832815 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.833610 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:26.833906 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:26.834013 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:26.836938 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:55943: BlockingRecv error: recv got EOF from 127.10.11.195:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:27.494954 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:27.489354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:44611)
0927 00:34:27.489635 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:27.489649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:27.489734 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:27.489956 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:27.489958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:27.490162 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:27.490255 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.490260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.491066 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.491069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.491646 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.491656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.492279 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.492282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.492383 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.492962 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:27.492983 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:27.494846 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
W20260927 00:34:27.495098 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:27.489410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44611 (local address 127.10.11.254:39259)
0927 00:34:27.489727 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:27.489731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:27.489744 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:34:27.489797 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:27.489799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:27.489844 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:27.489923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:27.490363 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.490969 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.491831 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.492154 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.492264 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.494965 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:44611: BlockingRecv error: recv got EOF from 127.10.11.193:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:27.652312 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:27.646588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:58571)
0927 00:34:27.646897 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:27.646916 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:27.647046 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:27.647306 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:27.647309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:27.647524 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:27.647619 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.647625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.648382 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.648387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.648949 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.648956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.649400 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.649403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.649488 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.649973 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:27.649984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:27.652205 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
W20260927 00:34:27.652416 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:27.646716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:58571 (local address 127.10.11.254:39259)
0927 00:34:27.646868 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:27.646872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:27.647008 (+ 136us) server_negotiation.cc:407] Connection header received
0927 00:34:27.647147 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:27.647151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:27.647204 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:27.647289 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:27.647736 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.648266 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.649065 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.649291 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.649400 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.652330 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:58571: BlockingRecv error: recv got EOF from 127.10.11.194:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:27 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.195@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:27.842762 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:27.837641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.195:45425)
0927 00:34:27.837888 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:27.837900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:27.838011 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:27.838305 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:27.838307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:27.838490 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:27.838581 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.838586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.839349 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.839352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.839900 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:27.839907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.840323 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.840325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:27.840385 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.840835 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:27.840846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:27.842648 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
W20260927 00:34:27.842960 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:27.837728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:45425 (local address 127.10.11.254:39259)
0927 00:34:27.837897 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:27.837902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:27.837971 (+ 69us) server_negotiation.cc:407] Connection header received
0927 00:34:27.838149 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:27.838151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:27.838199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:27.838272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:27.838712 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.839255 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.840009 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:27.840227 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:27.840306 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:27.842778 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.195:45425: BlockingRecv error: recv got EOF from 127.10.11.195:45425 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 00:34:28 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:28 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.193@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:28.500887 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:28.495889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.193:39965)
0927 00:34:28.496131 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:28.496144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:28.496233 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:28.496480 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:28.496483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:28.496649 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:28.496740 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:28.496745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.497514 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.497517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:28.498084 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:28.498092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.498526 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.498528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:28.498609 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:28.499063 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:28.499074 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:28.500761 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
W20260927 00:34:28.501006 19486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:28.495891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:39965 (local address 127.10.11.254:39259)
0927 00:34:28.496127 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:28.496130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:28.496213 (+ 83us) server_negotiation.cc:407] Connection header received
0927 00:34:28.496309 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:28.496311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:28.496353 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:28.496427 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:28.496837 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.497417 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.498207 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.498435 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.498509 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:28.500880 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.193:39965: BlockingRecv error: recv got EOF from 127.10.11.193:39965 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
Sep 27 00:34:28 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:34:28 dist-test-slave-6gxc krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.11.194@KRBTEST.COM for oryx/127.10.11.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:34:28.658776 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:28.653364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39259 (local address 127.10.11.194:47287)
0927 00:34:28.653680 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:28.653694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:28.653787 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:28.654017 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:28.654020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:28.654271 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:28.654382 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:28.654388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.655086 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.655089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:28.655697 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:28.655705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.656129 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.656132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:28.656209 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:28.656693 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:28.656704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:28.658609 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.11.254:39259: Server oryx/127.10.11.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":145,"threads_started":1}
W20260927 00:34:28.658879 19486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:28.653414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:47287 (local address 127.10.11.254:39259)
0927 00:34:28.653529 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:28.653532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:28.653763 (+ 231us) server_negotiation.cc:407] Connection header received
0927 00:34:28.653871 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:28.653873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:28.653911 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:28.653974 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:28.654481 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.654977 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.655804 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.656023 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.656099 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:28.658759 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.11.194:47287: BlockingRecv error: recv got EOF from 127.10.11.194:47287 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:34:28.742144 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19012
I20260927 00:34:28.747584 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19164
I20260927 00:34:28.752772 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19296
I20260927 00:34:28.757776 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 18942
2026-09-27T00:34:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17843 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:28Z Disabled control of system clock
I20260927 00:34:28.786993 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:35197
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:35197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:34:28.891052 19492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:28.891322 19492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:28.891393 19492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:28.895287 19492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:28.895367 19492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:28.895411 19492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:28.895449 19492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:28.895479 19492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:28.900522 19492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:35197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:28.901448 19492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:28.902746 19492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:28.911033 19500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:28.910995 19498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:28.911026 19492 server_base.cc:1034] running on GCE node
W20260927 00:34:28.910989 19497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:28.911765 19492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:28.912792 19492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:28.914004 19492 hybrid_clock.cc:648] HybridClock initialized: now 1790469268913990 us; error 36 us; skew 500 ppm
I20260927 00:34:28.915815 19492 webserver.cc:460] Webserver started at http://127.10.11.254:41927/ using document root <none> and password file <none>
I20260927 00:34:28.916390 19492 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:28.916481 19492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:28.916699 19492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:28.918680 19492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "93391f2fc222424084fb67152cb736de"
format_stamp: "Formatted at 2026-09-27 00:34:28 on dist-test-slave-6gxc"
server_key: "28b60e922576c5d20d490e917ea9ae20"
server_key_iv: "d60d3131f4ee2f2f924b6f104157dec8"
server_key_version: "encryptionkey@0"
I20260927 00:34:28.919199 19492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "93391f2fc222424084fb67152cb736de"
format_stamp: "Formatted at 2026-09-27 00:34:28 on dist-test-slave-6gxc"
server_key: "28b60e922576c5d20d490e917ea9ae20"
server_key_iv: "d60d3131f4ee2f2f924b6f104157dec8"
server_key_version: "encryptionkey@0"
I20260927 00:34:28.922564 19492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:28.924783 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:28.925889 19492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:28.926029 19492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "93391f2fc222424084fb67152cb736de"
format_stamp: "Formatted at 2026-09-27 00:34:28 on dist-test-slave-6gxc"
server_key: "28b60e922576c5d20d490e917ea9ae20"
server_key_iv: "d60d3131f4ee2f2f924b6f104157dec8"
server_key_version: "encryptionkey@0"
I20260927 00:34:28.926168 19492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:28.937397 19492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:28.937938 19492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:28.938072 19492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:28.944461 19492 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:35197
I20260927 00:34:28.944489 19558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:35197 every 8 connection(s)
I20260927 00:34:28.945310 19492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:28.948381 19559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:28.951632 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19492
I20260927 00:34:28.951764 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:28.952109 10287 external_mini_cluster.cc:1428] Setting key 029c24b80f5ceff8276324bb5483840a
I20260927 00:34:28.954310 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de: Bootstrap starting.
I20260927 00:34:28.956768 19559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:28.957500 19559 log.cc:824] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:28.959326 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de: No bootstrap required, opened a new log
I20260927 00:34:28.961771 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:28.953467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.10.11.254:35197)
0927 00:34:28.954051 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:28.954065 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:28.954140 (+ 75us) server_negotiation.cc:407] Connection header received
0927 00:34:28.955064 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:28.955087 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:28.955587 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:28.955901 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:28.956324 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.957373 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.958514 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:28.958979 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:28.959154 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:28.960004 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:28.960009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:28.960021 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:28.960375 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:28.960426 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:28.960445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:28.960574 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:28.960721 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:34:28.960956 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260927 00:34:28.962560 19559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } }
I20260927 00:34:28.962827 19559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:28.962920 19559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93391f2fc222424084fb67152cb736de, State: Initialized, Role: FOLLOWER
I20260927 00:34:28.963405 19559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } }
I20260927 00:34:28.963529 19559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:28.963620 19559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:28.963728 19559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:28.964641 19559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } }
I20260927 00:34:28.964920 19559 leader_election.cc:302] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93391f2fc222424084fb67152cb736de; no voters:
I20260927 00:34:28.965155 19559 leader_election.cc:288] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:28.965265 19564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:28.965480 19564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [term 1 LEADER]: Becoming Leader. State: Replica: 93391f2fc222424084fb67152cb736de, State: Running, Role: LEADER
I20260927 00:34:28.965802 19564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } }
I20260927 00:34:28.966391 19559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:28.967376 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93391f2fc222424084fb67152cb736de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } } }
I20260927 00:34:28.967499 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:28.967882 19573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:28.967829 19566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93391f2fc222424084fb67152cb736de. Latest consensus state: current_term: 1 leader_uuid: "93391f2fc222424084fb67152cb736de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93391f2fc222424084fb67152cb736de" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 35197 } } }
I20260927 00:34:28.967973 19566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:28.971010 19573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:28.976238 19573 catalog_manager.cc:1347] Generated new cluster ID: d571def5a3664cee98317df6167f2d5e
I20260927 00:34:28.976295 19573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:28.997159 19573 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:28.998311 19573 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:29.006438 19573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93391f2fc222424084fb67152cb736de: Generated new TSK 0
I20260927 00:34:29.007261 19573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:34:29.011545 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--builtin_ntp_servers=127.10.11.212:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:34:29.117105 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:29.117311 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:29.117344 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:29.121222 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:29.121265 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:29.121313 19583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:29.125738 19583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:29.126655 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:29.127871 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:29.136773 19583 server_base.cc:1034] running on GCE node
W20260927 00:34:29.136816 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.136817 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.136720 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.137411 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:29.137912 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:29.139058 19583 hybrid_clock.cc:648] HybridClock initialized: now 1790469269139045 us; error 62 us; skew 500 ppm
I20260927 00:34:29.140700 19583 webserver.cc:460] Webserver started at http://127.10.11.193:45549/ using document root <none> and password file <none>
I20260927 00:34:29.141224 19583 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:29.141281 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:29.141508 19583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:29.143373 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "2ee82c150e6649c9be2e039b871f4973"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e47e97a6ad4b358c163c3bcba570a00a"
server_key_iv: "ed79e7919b788d546d37342fe4731501"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.143885 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee82c150e6649c9be2e039b871f4973"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e47e97a6ad4b358c163c3bcba570a00a"
server_key_iv: "ed79e7919b788d546d37342fe4731501"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.147186 19583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:29.149343 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.150476 19583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:29.150597 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "2ee82c150e6649c9be2e039b871f4973"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e47e97a6ad4b358c163c3bcba570a00a"
server_key_iv: "ed79e7919b788d546d37342fe4731501"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.150698 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:29.184923 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:29.185747 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:29.185982 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:29.186738 19583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:29.187849 19583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:29.187920 19583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.188001 19583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:29.188033 19583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.196077 19583 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:34357
I20260927 00:34:29.196141 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:34357 every 8 connection(s)
I20260927 00:34:29.196934 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:29.198349 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19583
I20260927 00:34:29.198436 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:29.198719 10287 external_mini_cluster.cc:1428] Setting key ce54bd8c87611fa63c1611e18f5a8a20
I20260927 00:34:29.201085 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--builtin_ntp_servers=127.10.11.212:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:34:29.207696 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.200090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:58481 (local address 127.10.11.254:35197)
0927 00:34:29.200228 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:29.200232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:29.200248 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:34:29.201719 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:29.201722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:29.201767 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:29.201831 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:29.202931 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.203760 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.205559 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.205772 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.205849 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.207197 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:29.207201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:29.207204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:29.207217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:29.207237 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:29.207247 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:29.207318 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:29.207502 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:34:29.207568 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:34:29.208837 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.199082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35197 (local address 127.10.11.193:58481)
0927 00:34:29.199804 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:29.199841 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:29.201492 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:29.202033 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:29.202045 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:29.202193 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:29.202798 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.202811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.203884 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.203923 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.205436 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.205450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.205927 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.205935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.206195 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.206880 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:29.206884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:29.206951 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:29.207038 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:29.207341 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:29.207351 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:29.207359 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:34:29.207497 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:34:29.207809 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":161,"threads_started":1}
I20260927 00:34:29.210049 19711 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35197
I20260927 00:34:29.210278 19711 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:29.210721 19711 heartbeater.cc:511] Master 127.10.11.254:35197 requested a full tablet report, sending...
I20260927 00:34:29.212669 19523 ts_manager.cc:194] Registered new tserver with Master: 2ee82c150e6649c9be2e039b871f4973 (127.10.11.193:34357)
I20260927 00:34:29.214385 19523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.193:58481
W20260927 00:34:29.307655 19714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:29.307914 19714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:29.307984 19714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:29.311955 19714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:29.312023 19714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:29.312107 19714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:29.316684 19714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:29.317612 19714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:29.318969 19714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:29.328402 19714 server_base.cc:1034] running on GCE node
W20260927 00:34:29.328395 19722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.328352 19720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.328349 19719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.329020 19714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:29.329629 19714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:29.330796 19714 hybrid_clock.cc:648] HybridClock initialized: now 1790469269330780 us; error 27 us; skew 500 ppm
I20260927 00:34:29.332399 19714 webserver.cc:460] Webserver started at http://127.10.11.194:33285/ using document root <none> and password file <none>
I20260927 00:34:29.332880 19714 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:29.332934 19714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:29.333137 19714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:29.335076 19714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "933d6c3aff8d40aea21cf531c8c73335"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e31a2d38741c9543c9a91f141de02283"
server_key_iv: "1d1e28cf44076e8c0c7b32ea60f93db6"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.335603 19714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "933d6c3aff8d40aea21cf531c8c73335"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e31a2d38741c9543c9a91f141de02283"
server_key_iv: "1d1e28cf44076e8c0c7b32ea60f93db6"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.338979 19714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:29.341192 19728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.342502 19714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:29.342630 19714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "933d6c3aff8d40aea21cf531c8c73335"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "e31a2d38741c9543c9a91f141de02283"
server_key_iv: "1d1e28cf44076e8c0c7b32ea60f93db6"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.342732 19714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:29.356125 19714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:29.356684 19714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:29.356856 19714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:29.357393 19714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:29.358295 19714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:29.358340 19714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.358376 19714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:29.358390 19714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.367393 19714 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:33287
I20260927 00:34:29.367435 19841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:33287 every 8 connection(s)
I20260927 00:34:29.368269 19714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:29.376785 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.370063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:53277 (local address 127.10.11.254:35197)
0927 00:34:29.370223 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:29.370226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:29.371204 (+ 978us) server_negotiation.cc:407] Connection header received
0927 00:34:29.372464 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:29.372467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:29.372512 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:29.372586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:29.373420 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.373929 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.375120 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.375348 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.375444 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.376391 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:29.376394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:29.376405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:29.376417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:29.376435 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:29.376443 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:29.376502 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:29.376636 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:34:29.376682 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:34:29.377539 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.370447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35197 (local address 127.10.11.194:53277)
0927 00:34:29.371065 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:29.371102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:29.372275 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:29.372732 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:29.372738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:29.372814 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:29.373274 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.373293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.374041 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.374044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.374975 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.374985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.375500 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.375503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.375639 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.376163 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:29.376165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:29.376200 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:29.376257 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:29.376516 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:29.376521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:29.376525 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:34:29.376620 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:34:29.376812 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260927 00:34:29.377867 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19714
I20260927 00:34:29.377988 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:29.378413 10287 external_mini_cluster.cc:1428] Setting key c93007125e36bf69e383353e37ca08a9
I20260927 00:34:29.378660 19842 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35197
I20260927 00:34:29.378901 19842 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:29.379432 19842 heartbeater.cc:511] Master 127.10.11.254:35197 requested a full tablet report, sending...
I20260927 00:34:29.380384 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--builtin_ntp_servers=127.10.11.212:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:34:29.380416 19523 ts_manager.cc:194] Registered new tserver with Master: 933d6c3aff8d40aea21cf531c8c73335 (127.10.11.194:33287)
I20260927 00:34:29.380977 19523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.194:53277
W20260927 00:34:29.484522 19845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:29.484737 19845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:29.484771 19845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:29.489166 19845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:29.489219 19845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:29.489269 19845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:29.493848 19845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:29.494697 19845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:29.495950 19845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:29.504611 19851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.504657 19850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.504660 19845 server_base.cc:1034] running on GCE node
W20260927 00:34:29.504935 19853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.505352 19845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:29.505887 19845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:29.507066 19845 hybrid_clock.cc:648] HybridClock initialized: now 1790469269507033 us; error 44 us; skew 500 ppm
I20260927 00:34:29.512722 19845 webserver.cc:460] Webserver started at http://127.10.11.195:33923/ using document root <none> and password file <none>
I20260927 00:34:29.513240 19845 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:29.513331 19845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:29.513542 19845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:29.515534 19845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "719cacb853d14d0eb2a3d29727587882"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "747f4643a22785837f89726c263d47b8"
server_key_iv: "c34db9c2b6374762452519ae1e59f32c"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.516065 19845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "719cacb853d14d0eb2a3d29727587882"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "747f4643a22785837f89726c263d47b8"
server_key_iv: "c34db9c2b6374762452519ae1e59f32c"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.519353 19845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:29.521584 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.522742 19845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:29.522869 19845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "719cacb853d14d0eb2a3d29727587882"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "747f4643a22785837f89726c263d47b8"
server_key_iv: "c34db9c2b6374762452519ae1e59f32c"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.522974 19845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:29.543825 19845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:29.544453 19845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:29.544629 19845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:29.545136 19845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:29.546022 19845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:29.546087 19845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.546161 19845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:29.546198 19845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.554767 19845 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:38133
I20260927 00:34:29.554836 19972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:38133 every 8 connection(s)
I20260927 00:34:29.555583 19845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:29.556018 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19845
I20260927 00:34:29.556126 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:29.556434 10287 external_mini_cluster.cc:1428] Setting key 5e556c69880dafa955a358460c176d92
I20260927 00:34:29.564504 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.557503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34687 (local address 127.10.11.254:35197)
0927 00:34:29.557616 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:29.557624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:29.558908 (+ 1284us) server_negotiation.cc:407] Connection header received
0927 00:34:29.560200 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:29.560202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:29.560244 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:29.560309 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:29.561095 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.561597 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.562739 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.562952 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.563042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.564023 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:29.564026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:29.564028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:29.564038 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:29.564052 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:29.564058 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:29.564115 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:29.564349 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:34:29.564394 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:34:29.565330 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.558001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:35197 (local address 127.10.11.195:34687)
0927 00:34:29.558786 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:29.558823 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:29.560038 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:29.560471 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:29.560478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:29.560552 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:29.560989 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.561000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.561694 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.561696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.562614 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:29.562622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:29.563076 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:29.563078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:29.563207 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:29.563808 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:29.563809 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:29.563845 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:29.563902 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:29.564213 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:29.564221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:29.564225 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:34:29.564337 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:34:29.564591 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":111,"threads_started":1}
I20260927 00:34:29.566372 19973 heartbeater.cc:347] Connected to a master server at 127.10.11.254:35197
I20260927 00:34:29.566587 19973 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:29.567078 19973 heartbeater.cc:511] Master 127.10.11.254:35197 requested a full tablet report, sending...
I20260927 00:34:29.568091 19523 ts_manager.cc:194] Registered new tserver with Master: 719cacb853d14d0eb2a3d29727587882 (127.10.11.195:38133)
I20260927 00:34:29.568616 19523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.195:34687
I20260927 00:34:29.570266 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:34:29.572799 19562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:29.572003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.10.11.254:35197)
0927 00:34:29.572109 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:29.572113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:29.572309 (+ 196us) server_negotiation.cc:407] Connection header received
0927 00:34:29.572405 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:29.572407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:29.572438 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:29.572512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:29.572683 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33204: BlockingRecv error: recv got EOF from 127.0.0.1:33204 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:34:29.573240 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19583
I20260927 00:34:29.578850 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19714
I20260927 00:34:29.584195 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19845
I20260927 00:34:29.589283 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19492
2026-09-27T00:34:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:29Z Disabled control of system clock
I20260927 00:34:29.617061 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:46389
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:34:29.720963 19984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:29.721199 19984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:29.721235 19984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:29.725178 19984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:29.725229 19984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:29.725244 19984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:29.725260 19984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:29.725275 19984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:29.729787 19984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:46389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:29.730681 19984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:29.731942 19984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:29.739869 19992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.739863 19989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.740005 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.740185 19984 server_base.cc:1034] running on GCE node
I20260927 00:34:29.740731 19984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:29.741711 19984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:29.742875 19984 hybrid_clock.cc:648] HybridClock initialized: now 1790469269742842 us; error 44 us; skew 500 ppm
I20260927 00:34:29.744599 19984 webserver.cc:460] Webserver started at http://127.10.11.254:43873/ using document root <none> and password file <none>
I20260927 00:34:29.745128 19984 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:29.745186 19984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:29.745436 19984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:29.747381 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "7da50c61b5a24e5b86e8505aaa9b128c"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "01531af4bc32cdbf8e211a00a5628ac6"
server_key_iv: "fbf1af499e7b6822640ee9ebac8e5211"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.747948 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "7da50c61b5a24e5b86e8505aaa9b128c"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "01531af4bc32cdbf8e211a00a5628ac6"
server_key_iv: "fbf1af499e7b6822640ee9ebac8e5211"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.751427 19984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:29.753597 19998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:29.754776 19984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:29.754917 19984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "7da50c61b5a24e5b86e8505aaa9b128c"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "01531af4bc32cdbf8e211a00a5628ac6"
server_key_iv: "fbf1af499e7b6822640ee9ebac8e5211"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.755024 19984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:29.759732 19984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:29.760281 19984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:29.760459 19984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:29.766989 19984 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:46389
I20260927 00:34:29.767021 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:46389 every 8 connection(s)
I20260927 00:34:29.767822 19984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:29.770712 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:29.771767 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 19984
I20260927 00:34:29.771921 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:29.772202 10287 external_mini_cluster.cc:1428] Setting key 2b7930de9618e795a40b302a8f48a0ec
I20260927 00:34:29.776485 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c: Bootstrap starting.
I20260927 00:34:29.777421 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:29.773437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.10.11.254:46389)
0927 00:34:29.774008 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:29.774023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:29.774056 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:34:29.774710 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:29.774725 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:29.775039 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:29.775297 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:29.776196 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:29.776200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:29.776211 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:29.776222 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:29.776276 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:29.776291 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:29.776418 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:29.776520 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:34:29.776762 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
I20260927 00:34:29.778709 20051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:29.779534 20051 log.cc:824] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:29.781174 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c: No bootstrap required, opened a new log
I20260927 00:34:29.784730 20051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } }
I20260927 00:34:29.784921 20051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:29.784960 20051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da50c61b5a24e5b86e8505aaa9b128c, State: Initialized, Role: FOLLOWER
I20260927 00:34:29.785287 20051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } }
I20260927 00:34:29.785387 20051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:29.785423 20051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:29.785488 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:29.786338 20051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } }
I20260927 00:34:29.786604 20051 leader_election.cc:302] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da50c61b5a24e5b86e8505aaa9b128c; no voters:
I20260927 00:34:29.786867 20051 leader_election.cc:288] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:29.787282 20056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:29.787488 20056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [term 1 LEADER]: Becoming Leader. State: Replica: 7da50c61b5a24e5b86e8505aaa9b128c, State: Running, Role: LEADER
I20260927 00:34:29.787814 20056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } }
I20260927 00:34:29.788208 20051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:29.789446 20057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } } }
I20260927 00:34:29.789454 20058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da50c61b5a24e5b86e8505aaa9b128c. Latest consensus state: current_term: 1 leader_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da50c61b5a24e5b86e8505aaa9b128c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46389 } } }
I20260927 00:34:29.789597 20057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:29.789600 20058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:29.789986 20065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:29.792914 20065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:29.797994 20065 catalog_manager.cc:1347] Generated new cluster ID: 3404386f86bf4ba6a50296ccf4658bb7
I20260927 00:34:29.798065 20065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:29.832858 20065 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:29.833169 20065 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:29.849454 20065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da50c61b5a24e5b86e8505aaa9b128c: Generated new TSK 0
I20260927 00:34:29.850245 20065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:34:29.862620 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--builtin_ntp_servers=127.10.11.212:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:34:29.967783 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:29.968041 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:29.968120 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:29.971972 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:29.972054 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:29.972143 20075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:29.976658 20075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:29.977602 20075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:29.978875 20075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:29.987715 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.987738 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:29.987821 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:29.988253 20075 server_base.cc:1034] running on GCE node
I20260927 00:34:29.988684 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:29.989315 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:29.990507 20075 hybrid_clock.cc:648] HybridClock initialized: now 1790469269990502 us; error 50 us; skew 500 ppm
I20260927 00:34:29.992251 20075 webserver.cc:460] Webserver started at http://127.10.11.193:45017/ using document root <none> and password file <none>
I20260927 00:34:29.992779 20075 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:29.992874 20075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:29.993093 20075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:29.995042 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "ba4d02d5f00b42abb871da49be0bd655"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "cba8e4e3bb999d834126f81109e74659"
server_key_iv: "ffe93dc3f22524062bf836c8e79e2f42"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.995576 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "ba4d02d5f00b42abb871da49be0bd655"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "cba8e4e3bb999d834126f81109e74659"
server_key_iv: "ffe93dc3f22524062bf836c8e79e2f42"
server_key_version: "encryptionkey@0"
I20260927 00:34:29.998991 20075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:30.001204 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.002719 20075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:30.002835 20075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "ba4d02d5f00b42abb871da49be0bd655"
format_stamp: "Formatted at 2026-09-27 00:34:29 on dist-test-slave-6gxc"
server_key: "cba8e4e3bb999d834126f81109e74659"
server_key_iv: "ffe93dc3f22524062bf836c8e79e2f42"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.002956 20075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:30.007161 20075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:30.007716 20075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:30.007896 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:30.008411 20075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:30.009213 20075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:30.009280 20075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.009343 20075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:30.009407 20075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.017555 20075 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:38037
I20260927 00:34:30.017620 20202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:38037 every 8 connection(s)
I20260927 00:34:30.018393 20075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:30.023983 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.020056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:51839 (local address 127.10.11.254:46389)
0927 00:34:30.020199 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:30.020203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:30.021114 (+ 911us) server_negotiation.cc:407] Connection header received
0927 00:34:30.022489 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:30.022492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:30.022528 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:30.022600 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:30.023609 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:30.023611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:30.023613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:30.023625 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:30.023644 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:30.023651 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:30.023723 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:30.023845 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:34:30.023886 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:34:30.024745 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.020393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46389 (local address 127.10.11.193:51839)
0927 00:34:30.020991 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:30.021029 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:30.022278 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:30.022754 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:30.022760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:30.022847 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:30.023369 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:30.023370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:30.023406 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:30.023468 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:30.023730 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:30.023736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:30.023740 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:34:30.023893 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:34:30.024061 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260927 00:34:30.025862 20203 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46389
I20260927 00:34:30.026147 20203 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:30.026544 20203 heartbeater.cc:511] Master 127.10.11.254:46389 requested a full tablet report, sending...
I20260927 00:34:30.027935 20015 ts_manager.cc:194] Registered new tserver with Master: ba4d02d5f00b42abb871da49be0bd655 (127.10.11.193:38037)
I20260927 00:34:30.028297 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20075
I20260927 00:34:30.028409 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:30.028712 10287 external_mini_cluster.cc:1428] Setting key e182cec991b3b7a96b0cd23b23cd6c73
I20260927 00:34:30.030623 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--builtin_ntp_servers=127.10.11.212:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:34:30.135669 20206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:30.135900 20206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:30.135977 20206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:30.140137 20206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:30.140187 20206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:30.140276 20206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:30.144878 20206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:30.145756 20206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:30.147017 20206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:30.155990 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:30.156039 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:30.156167 20206 server_base.cc:1034] running on GCE node
W20260927 00:34:30.156196 20214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:30.156595 20206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:30.157217 20206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:30.158445 20206 hybrid_clock.cc:648] HybridClock initialized: now 1790469270158416 us; error 36 us; skew 500 ppm
I20260927 00:34:30.160241 20206 webserver.cc:460] Webserver started at http://127.10.11.194:40379/ using document root <none> and password file <none>
I20260927 00:34:30.160743 20206 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:30.160806 20206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:30.160992 20206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:30.162846 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "60cf4ea23d974d54a333ca79722ab087"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "2999ac8238d9ca0a2801588a0b39cd92"
server_key_iv: "48301ae43542357819f7fcfb1f0597f9"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.163362 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "60cf4ea23d974d54a333ca79722ab087"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "2999ac8238d9ca0a2801588a0b39cd92"
server_key_iv: "48301ae43542357819f7fcfb1f0597f9"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.166574 20206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:30.168825 20220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.169932 20206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:34:30.170065 20206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "60cf4ea23d974d54a333ca79722ab087"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "2999ac8238d9ca0a2801588a0b39cd92"
server_key_iv: "48301ae43542357819f7fcfb1f0597f9"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.170217 20206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:30.174439 20206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:30.174962 20206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:30.175159 20206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:30.175633 20206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:30.176432 20206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:30.176508 20206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.176569 20206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:30.176621 20206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.185257 20206 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:45205
I20260927 00:34:30.185279 20333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:45205 every 8 connection(s)
I20260927 00:34:30.186049 20206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:34:30.191867 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.187696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:38617 (local address 127.10.11.254:46389)
0927 00:34:30.187841 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:30.187846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:30.188981 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 00:34:30.190396 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:30.190398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:30.190432 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:30.190510 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:30.191537 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:30.191539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:30.191540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:30.191548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:30.191561 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:30.191567 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:30.191630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:30.191739 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:34:30.191771 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:34:30.192626 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.188150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46389 (local address 127.10.11.194:38617)
0927 00:34:30.188853 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:30.188892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:30.190223 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:30.190647 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:30.190653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:30.190731 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:30.191301 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:30.191302 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:30.191338 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:30.191411 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:30.191633 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:30.191637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:30.191641 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:34:30.191752 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:34:30.191915 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260927 00:34:30.193588 20334 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46389
I20260927 00:34:30.193792 20334 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:30.194226 20334 heartbeater.cc:511] Master 127.10.11.254:46389 requested a full tablet report, sending...
I20260927 00:34:30.195161 20015 ts_manager.cc:194] Registered new tserver with Master: 60cf4ea23d974d54a333ca79722ab087 (127.10.11.194:45205)
I20260927 00:34:30.196096 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20206
I20260927 00:34:30.196223 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:30.196512 10287 external_mini_cluster.cc:1428] Setting key 03b386a812f3e020022b72a02113e7b8
I20260927 00:34:30.198266 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--builtin_ntp_servers=127.10.11.212:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:34:30.301664 20337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:30.301900 20337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:30.301973 20337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:30.305908 20337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:30.305972 20337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:30.306052 20337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:30.310750 20337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:30.311604 20337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:30.312778 20337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:30.321587 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:30.321708 20337 server_base.cc:1034] running on GCE node
W20260927 00:34:30.321587 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:30.321610 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:30.322237 20337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:30.322866 20337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:30.324057 20337 hybrid_clock.cc:648] HybridClock initialized: now 1790469270324043 us; error 28 us; skew 500 ppm
I20260927 00:34:30.325803 20337 webserver.cc:460] Webserver started at http://127.10.11.195:34451/ using document root <none> and password file <none>
I20260927 00:34:30.326424 20337 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:30.326512 20337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:30.326728 20337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:30.328615 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "37dd48cb3f43491ebd242fbb96a22c0c"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "3168d4207157ddaf4f056d4a865017ff"
server_key_iv: "ac3163b410b0c33540fcaf2f7db0cb22"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.329151 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "37dd48cb3f43491ebd242fbb96a22c0c"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "3168d4207157ddaf4f056d4a865017ff"
server_key_iv: "ac3163b410b0c33540fcaf2f7db0cb22"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.332540 20337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:34:30.334952 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.336018 20337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:30.336146 20337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "37dd48cb3f43491ebd242fbb96a22c0c"
format_stamp: "Formatted at 2026-09-27 00:34:30 on dist-test-slave-6gxc"
server_key: "3168d4207157ddaf4f056d4a865017ff"
server_key_iv: "ac3163b410b0c33540fcaf2f7db0cb22"
server_key_version: "encryptionkey@0"
I20260927 00:34:30.336252 20337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:30.340545 20337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:30.341092 20337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:30.341264 20337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:30.341751 20337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:30.342633 20337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:30.342702 20337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.342770 20337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:30.342813 20337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:30.351152 20337 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:46269
I20260927 00:34:30.351171 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:46269 every 8 connection(s)
I20260927 00:34:30.351919 20337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:30.353283 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20337
I20260927 00:34:30.353420 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:30.353708 10287 external_mini_cluster.cc:1428] Setting key 1b42fe0a5b7df785652f4760ac7a3dd5
I20260927 00:34:30.357621 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.353591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:43763 (local address 127.10.11.254:46389)
0927 00:34:30.353712 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:30.353716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:30.354950 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 00:34:30.356231 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:30.356233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:30.356267 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:30.356332 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:30.357250 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:30.357253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:30.357255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:34:30.357266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:30.357285 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:34:30.357293 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:30.357385 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:30.357490 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:34:30.357525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:34:30.358423 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:30.354042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46389 (local address 127.10.11.195:43763)
0927 00:34:30.354790 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:30.354845 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:30.356059 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:30.356443 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:30.356448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:30.356530 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:30.357008 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:34:30.357009 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:30.357042 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:34:30.357101 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:30.357376 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:30.357381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:30.357385 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:34:30.357507 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:34:30.357692 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":117,"threads_started":1}
I20260927 00:34:30.359367 20465 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46389
I20260927 00:34:30.359573 20465 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:30.359954 20465 heartbeater.cc:511] Master 127.10.11.254:46389 requested a full tablet report, sending...
I20260927 00:34:30.360911 20015 ts_manager.cc:194] Registered new tserver with Master: 37dd48cb3f43491ebd242fbb96a22c0c (127.10.11.195:46269)
I20260927 00:34:30.367312 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:34:30.369974 20054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:34:30.369214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.10.11.254:46389)
0927 00:34:30.369347 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:30.369350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:30.369476 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:34:30.369551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:30.369553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:30.369578 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:30.369647 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:30.369810 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43456: BlockingRecv error: recv got EOF from 127.0.0.1:43456 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:34:30.370185 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20075
I20260927 00:34:30.375694 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20206
I20260927 00:34:30.381058 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20337
I20260927 00:34:30.386268 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 19984
2026-09-27T00:34:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (797 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:30 dist-test-slave-6gxc krb5kdc[20475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:30 dist-test-slave-6gxc krb5kdc[20475](info): set up 2 sockets
Sep 27 00:34:30 dist-test-slave-6gxc krb5kdc[20475](info): commencing operation
krb5kdc: starting...
W20260927 00:34:31.411278 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:31 dist-test-slave-6gxc krb5kdc[20475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:34:31.526360 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36343
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:31.630867 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:31.631130 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:31.631197 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:31.635177 20488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:34:31.635249 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:31.635288 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:31.635345 20488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:34:31.635382 20488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:34:31.640291 20488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:31.641202 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:31.642164 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:31.650161 20488 server_base.cc:1034] running on GCE node
W20260927 00:34:31.650113 20493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:31.650136 20494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:31.650122 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:31.650837 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:31.651789 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:31.652947 20488 hybrid_clock.cc:648] HybridClock initialized: now 1790469271652937 us; error 26 us; skew 500 ppm
Sep 27 00:34:31 dist-test-slave-6gxc krb5kdc[20475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:31.655534 20488 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:34:31.656355 20488 webserver.cc:460] Webserver started at http://127.10.11.254:41411/ using document root <none> and password file <none>
I20260927 00:34:31.656857 20488 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:31.656913 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:31.657126 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:31.659049 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "b72b4f03bd2a48b5950c9acfe34e181e"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "2259bb9ec353d79ec6d203143084e2a0"
server_key_iv: "04c11ed3a9f4fc6d632de52b99c4b71d"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.659569 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "b72b4f03bd2a48b5950c9acfe34e181e"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "2259bb9ec353d79ec6d203143084e2a0"
server_key_iv: "04c11ed3a9f4fc6d632de52b99c4b71d"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.662961 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:31.665212 20503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:31.666493 20488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:31.666622 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "b72b4f03bd2a48b5950c9acfe34e181e"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "2259bb9ec353d79ec6d203143084e2a0"
server_key_iv: "04c11ed3a9f4fc6d632de52b99c4b71d"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.666726 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:31.687903 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:31.690677 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:31.690864 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:31.697886 20488 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:36343
I20260927 00:34:31.698225 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:36343 every 8 connection(s)
I20260927 00:34:31.698812 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:34:31.701368 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:31.701393 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20488
I20260927 00:34:31.701535 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:34:31.701814 10287 external_mini_cluster.cc:1428] Setting key 087391b4e979fdb4ecf8293e1aaec88a
I20260927 00:34:31.707863 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: Bootstrap starting.
I20260927 00:34:31.710425 20556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: Neither blocks nor log segments found. Creating new log.
Sep 27 00:34:31 dist-test-slave-6gxc krb5kdc[20475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:31.711206 20556 log.cc:824] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:31.713336 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: No bootstrap required, opened a new log
I20260927 00:34:31.716142 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:31.703063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.10.11.254:36343)
0927 00:34:31.703439 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:31.703453 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:31.703485 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:34:31.704352 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:31.704369 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:31.704665 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:31.704921 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:31.705849 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.706832 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.708047 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.708472 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.708586 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:31.711199 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:31.711205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:31.711221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:31.711254 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:31.713153 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:31.713595 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:31.713601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:31.713606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:31.713749 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:31.714004 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:31.714007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:31.714009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:31.714506 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:31.714699 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:31.714885 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:34:31.715160 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
I20260927 00:34:31.717742 20556 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } }
I20260927 00:34:31.717993 20556 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:31.718087 20556 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72b4f03bd2a48b5950c9acfe34e181e, State: Initialized, Role: FOLLOWER
I20260927 00:34:31.718537 20556 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } }
I20260927 00:34:31.718654 20556 raft_consensus.cc:388] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:34:31.718735 20556 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:34:31.718825 20556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:31.719656 20556 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } }
I20260927 00:34:31.719985 20556 leader_election.cc:302] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b72b4f03bd2a48b5950c9acfe34e181e; no voters:
I20260927 00:34:31.720265 20556 leader_election.cc:288] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:34:31.720402 20561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:31.720607 20561 raft_consensus.cc:686] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [term 1 LEADER]: Becoming Leader. State: Replica: b72b4f03bd2a48b5950c9acfe34e181e, State: Running, Role: LEADER
I20260927 00:34:31.720868 20561 consensus_queue.cc:237] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } }
I20260927 00:34:31.721400 20556 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:34:31.721880 20561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b72b4f03bd2a48b5950c9acfe34e181e. Latest consensus state: current_term: 1 leader_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } } }
I20260927 00:34:31.722024 20561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:31.722460 20566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:34:31.722436 20562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72b4f03bd2a48b5950c9acfe34e181e" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36343 } } }
I20260927 00:34:31.722630 20562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e [sys.catalog]: This master's current role is: LEADER
I20260927 00:34:31.725358 20566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:34:31.731081 20566 catalog_manager.cc:1347] Generated new cluster ID: 20904fbb438f40c4a89c5e2efa615f77
I20260927 00:34:31.731140 20566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:34:31.738075 20566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:34:31.739128 20566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:34:31.745553 20566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: Generated new TSK 0
I20260927 00:34:31.746156 20566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:34:31.776724 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:31.899670 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:31.899892 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:31.899931 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:31.903993 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:31.904040 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:31.904122 20582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:34:31.909320 20582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:31.910264 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:31.911170 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:31.920198 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:31.920225 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:31.920209 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:31.920318 20582 server_base.cc:1034] running on GCE node
I20260927 00:34:31.920794 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:31.921353 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:31.922534 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790469271922521 us; error 25 us; skew 500 ppm
Sep 27 00:34:31 dist-test-slave-6gxc krb5kdc[20475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:31.925089 20582 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:34:31.925840 20582 webserver.cc:460] Webserver started at http://127.10.11.193:43917/ using document root <none> and password file <none>
I20260927 00:34:31.926402 20582 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:31.926486 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:31.926700 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:31.928524 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "fcb4097bf6734a6182ce56161e8c6aea"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "92e67f159712c438285b46ad723bbee6"
server_key_iv: "bb5e0a9ccaf3f3484b8f8053eb98c27f"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.929046 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "fcb4097bf6734a6182ce56161e8c6aea"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "92e67f159712c438285b46ad723bbee6"
server_key_iv: "bb5e0a9ccaf3f3484b8f8053eb98c27f"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.932413 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:34:31.934638 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:31.935706 20582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:31.935839 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "fcb4097bf6734a6182ce56161e8c6aea"
format_stamp: "Formatted at 2026-09-27 00:34:31 on dist-test-slave-6gxc"
server_key: "92e67f159712c438285b46ad723bbee6"
server_key_iv: "bb5e0a9ccaf3f3484b8f8053eb98c27f"
server_key_version: "encryptionkey@0"
I20260927 00:34:31.935941 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:31.948165 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:31.952618 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:31.953037 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:31.953751 20582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:31.954881 20582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:31.954926 20582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:31.954962 20582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:31.954982 20582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:31.966814 20582 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:46039
I20260927 00:34:31.966833 20710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:46039 every 8 connection(s)
I20260927 00:34:31.967907 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:34:31.972693 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20582
I20260927 00:34:31.972822 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:34:31.973151 10287 external_mini_cluster.cc:1428] Setting key b8cc553fbd38ee1202716c87581194cc
Sep 27 00:34:31 dist-test-slave-6gxc krb5kdc[20475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:34:31.984994 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:31.969925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:42431 (local address 127.10.11.254:36343)
0927 00:34:31.970049 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:31.970055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:31.971354 (+ 1299us) server_negotiation.cc:407] Connection header received
0927 00:34:31.972915 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:31.972919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:31.972979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:31.973031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:31.974716 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.975282 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.976516 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.976933 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.977041 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:31.980302 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:31.980306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:31.980308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:31.980335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:31.982313 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:31.982925 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:31.982931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:31.982935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:31.983035 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:31.983489 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:31.983494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:31.983497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:31.983763 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:31.983860 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:31.984678 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 00:34:31.984815 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:34:31.986449 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:31.970500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36343 (local address 127.10.11.193:42431)
0927 00:34:31.971150 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:31.971215 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:31.972669 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:31.973395 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:31.973401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:31.973852 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:31.974545 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:31.974565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.975430 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.975433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:31.976377 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:31.976387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:31.977076 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:31.977080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:31.977304 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:31.978000 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:31.978023 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:31.980132 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:31.982426 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:31.982432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:31.982447 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:31.982814 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:31.983159 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:31.983165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:31.983169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:31.983367 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:31.983971 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:31.983979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:31.984320 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 00:34:31.984598 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 00:34:31.984990 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":139,"threads_started":1}
I20260927 00:34:31.987996 20711 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36343
I20260927 00:34:31.988238 20711 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:31.988757 20711 heartbeater.cc:511] Master 127.10.11.254:36343 requested a full tablet report, sending...
I20260927 00:34:31.990685 20520 ts_manager.cc:194] Registered new tserver with Master: fcb4097bf6734a6182ce56161e8c6aea (127.10.11.193:46039)
I20260927 00:34:31.992319 20520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:42431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:34:32.001451 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:32.107216 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:32.107438 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:32.107477 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:32.111354 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:32.111399 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:32.111454 20716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:34:32.116343 20716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:32.117223 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:32.118185 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:34:32.127183 20716 server_base.cc:1034] running on GCE node
W20260927 00:34:32.127395 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:32.127444 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:32.127456 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:32.127873 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:32.128409 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:32.129595 20716 hybrid_clock.cc:648] HybridClock initialized: now 1790469272129583 us; error 22 us; skew 500 ppm
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[20475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:32.132226 20716 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:34:32.133090 20716 webserver.cc:460] Webserver started at http://127.10.11.194:35629/ using document root <none> and password file <none>
I20260927 00:34:32.133615 20716 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:32.133675 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:32.133913 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:32.135836 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "57fdd498923e49fb976f8668cf002b6f"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "85c3e3eff6fb779c6f051ece13773681"
server_key_iv: "bbd245401f897368f88d44cbfb67abe5"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.136391 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "57fdd498923e49fb976f8668cf002b6f"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "85c3e3eff6fb779c6f051ece13773681"
server_key_iv: "bbd245401f897368f88d44cbfb67abe5"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.139906 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:34:32.142062 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.143224 20716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:32.143370 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "57fdd498923e49fb976f8668cf002b6f"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "85c3e3eff6fb779c6f051ece13773681"
server_key_iv: "bbd245401f897368f88d44cbfb67abe5"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.143474 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:32.153525 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:32.156047 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:32.156232 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:32.156751 20716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:32.157490 20716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:32.157559 20716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.157616 20716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:32.157667 20716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.166054 20716 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:34971
I20260927 00:34:32.166081 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:34971 every 8 connection(s)
I20260927 00:34:32.166851 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[20475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:32.176918 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20716
I20260927 00:34:32.177050 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:34:32.177332 10287 external_mini_cluster.cc:1428] Setting key afe9c9c5dcd15db6452f34e4395d1cab
I20260927 00:34:32.179596 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.168442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:33897 (local address 127.10.11.254:36343)
0927 00:34:32.168565 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.168569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.169549 (+ 980us) server_negotiation.cc:407] Connection header received
0927 00:34:32.170551 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.170554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.170611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.170687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:32.171780 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.172303 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.173393 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.173617 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.173723 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.175904 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:32.175907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:32.175909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:32.175929 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:32.177335 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.178043 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.178047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.178048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.178158 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.178531 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.178533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.178534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.178749 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:32.178863 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.179323 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.179436 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:34:32.180353 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.168781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36343 (local address 127.10.11.194:33897)
0927 00:34:32.169416 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.169453 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.170308 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.170847 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.170854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.171170 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:32.171633 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.171651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.172426 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.172430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.173267 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.173276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.173757 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.173761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.173925 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.174420 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:32.174434 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:32.175760 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:32.177447 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:32.177453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:32.177466 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:32.177924 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:32.178260 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:32.178263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:32.178265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:32.178433 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:32.178860 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:32.178865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:32.179084 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.179268 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.179572 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":108,"threads_started":1}
I20260927 00:34:32.181574 20845 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36343
I20260927 00:34:32.181805 20845 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:32.182281 20845 heartbeater.cc:511] Master 127.10.11.254:36343 requested a full tablet report, sending...
I20260927 00:34:32.183334 20520 ts_manager.cc:194] Registered new tserver with Master: 57fdd498923e49fb976f8668cf002b6f (127.10.11.194:34971)
I20260927 00:34:32.184170 20520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:33897
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:34:32.203047 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:34:32.307972 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:34:32.308182 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:34:32.308221 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:34:32.312024 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:34:32.312067 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:34:32.312120 20850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:34:32.316926 20850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:34:32.317780 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:34:32.318667 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:34:32.327310 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:32.327404 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:34:32.327327 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:34:32.327710 20850 server_base.cc:1034] running on GCE node
I20260927 00:34:32.328059 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:34:32.328555 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:34:32.329732 20850 hybrid_clock.cc:648] HybridClock initialized: now 1790469272329724 us; error 30 us; skew 500 ppm
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[20475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:34:32.332235 20850 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:34:32.333009 20850 webserver.cc:460] Webserver started at http://127.10.11.195:36223/ using document root <none> and password file <none>
I20260927 00:34:32.333458 20850 fs_manager.cc:359] Metadata directory not provided
I20260927 00:34:32.333513 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:34:32.333658 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:34:32.335464 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "f322ecf8290a41bd867c83125a69e220"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "126b91d842aa205f0db8b4caa2d26c34"
server_key_iv: "2d5b173fc0c0f0a64ee10cbb4f5aaaf0"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.335922 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "f322ecf8290a41bd867c83125a69e220"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "126b91d842aa205f0db8b4caa2d26c34"
server_key_iv: "2d5b173fc0c0f0a64ee10cbb4f5aaaf0"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.339246 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:34:32.341401 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.342548 20850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:34:32.342634 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "f322ecf8290a41bd867c83125a69e220"
format_stamp: "Formatted at 2026-09-27 00:34:32 on dist-test-slave-6gxc"
server_key: "126b91d842aa205f0db8b4caa2d26c34"
server_key_iv: "2d5b173fc0c0f0a64ee10cbb4f5aaaf0"
server_key_version: "encryptionkey@0"
I20260927 00:34:32.342707 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:34:32.358291 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:34:32.360641 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:34:32.360769 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:34:32.361198 20850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:34:32.361847 20850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:34:32.361896 20850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.361930 20850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:34:32.361953 20850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:34:32.371069 20850 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:39301
I20260927 00:34:32.371114 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:39301 every 8 connection(s)
I20260927 00:34:32.371845 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:34:32.378224 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 20850
I20260927 00:34:32.378335 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:34:32.378657 10287 external_mini_cluster.cc:1428] Setting key 3841bbf268800a7527929ee088f8461e
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[20475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:34:32.384536 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.373462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:47667 (local address 127.10.11.254:36343)
0927 00:34:32.373591 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.373595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.374380 (+ 785us) server_negotiation.cc:407] Connection header received
0927 00:34:32.375624 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.375627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.375672 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.375745 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:32.376832 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.377318 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.378735 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.379001 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.379118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.381612 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:32.381615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:32.381617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:32.381639 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:32.382915 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.383333 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.383335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.383336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.383404 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.383697 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.383699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.383700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.383847 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:32.383919 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.384323 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.384403 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:34:32.385214 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.373789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36343 (local address 127.10.11.195:47667)
0927 00:34:32.374251 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.374288 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.375441 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.375932 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.375962 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.376281 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:34:32.376711 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.376723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.377446 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.377448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.378570 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.378582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.379174 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.379178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.379359 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.380006 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:34:32.380026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:34:32.381473 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:34:32.383012 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:32.383016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:32.383025 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:32.383251 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:32.383502 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:34:32.383504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:34:32.383505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:34:32.383622 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:34:32.383929 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:34:32.383933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:34:32.384127 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.384281 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.384518 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260927 00:34:32.386251 20979 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36343
I20260927 00:34:32.386463 20979 heartbeater.cc:464] Registering TS with master...
I20260927 00:34:32.386879 20979 heartbeater.cc:511] Master 127.10.11.254:36343 requested a full tablet report, sending...
I20260927 00:34:32.387785 20520 ts_manager.cc:194] Registered new tserver with Master: f322ecf8290a41bd867c83125a69e220 (127.10.11.195:39301)
I20260927 00:34:32.388276 20520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:47667
I20260927 00:34:32.392094 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:34:32.400141 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.393991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.10.11.254:36343)
0927 00:34:32.394092 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.394095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.394288 (+ 193us) server_negotiation.cc:407] Connection header received
0927 00:34:32.394370 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.394372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.394417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.394479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:34:32.395037 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.395491 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.396230 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.396418 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.396486 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.397607 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:34:32.397610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:34:32.397611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:34:32.397631 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:34:32.398863 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.399229 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.399232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.399233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.399298 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:34:32.399518 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:34:32.399520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:34:32.399521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:34:32.399702 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:34:32.399798 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.399966 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.400043 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:34:32.402609 20520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:34:32.405339 20520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:34:32.418494 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.412892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:34971 (local address 127.0.0.1:36546)
0927 00:34:32.413491 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.413541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.413659 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.414243 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.414250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.414278 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:32.414404 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.414411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.415736 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.415743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.416913 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.416922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.417946 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.417950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.418049 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.418080 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.418242 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.418353 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":86,"threads_started":1}
I20260927 00:34:32.418502 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.413015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.10.11.194:34971)
0927 00:34:32.413729 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.413742 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.413768 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:34:32.413842 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.413846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.414043 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.414556 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:32.414702 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.415613 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.417046 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.417823 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.417949 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.418084 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.418269 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.418356 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":105,"threads_started":1}
I20260927 00:34:32.418812 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.412985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:39301 (local address 127.0.0.1:52692)
0927 00:34:32.413866 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.413881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.413947 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.414737 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.414740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.414756 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:32.414848 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.414856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.416212 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.416215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.417332 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.417341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.418425 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.418431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.418525 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.418558 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.418608 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.418679 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260927 00:34:32.419052 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.413569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46039 (local address 127.0.0.1:45704)
0927 00:34:32.414174 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.414187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.414265 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.414988 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.414991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.415007 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:32.415086 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.415092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.416446 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.416449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.417452 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.417462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.418686 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.418689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.418770 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.418789 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.418858 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.418923 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":32,"thread_start_us":84,"threads_started":1}
I20260927 00:34:32.419380 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.413319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.10.11.195:39301)
0927 00:34:32.414386 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.414397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.414412 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:34:32.414463 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.414466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.414608 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.414757 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:32.414971 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.416075 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.417465 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.418285 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.418973 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.419106 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.419157 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.419243 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":123,"threads_started":1}
I20260927 00:34:32.419420 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.413954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.10.11.193:46039)
0927 00:34:32.414377 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.414389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.414411 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:34:32.414484 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.414492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.414679 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.414820 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:32.415220 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.416300 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.417590 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.418406 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.418495 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.418629 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.419195 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.419274 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":106,"threads_started":1}
I20260927 00:34:32.421537 20779 tablet_service.cc:1467] Processing CreateTablet for tablet c741918c906544ca9f766bd0901d8985 (DEFAULT_TABLE table=test-table [id=aa5509d7068c4e78ac8254fe45561095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:32.421844 20913 tablet_service.cc:1467] Processing CreateTablet for tablet c741918c906544ca9f766bd0901d8985 (DEFAULT_TABLE table=test-table [id=aa5509d7068c4e78ac8254fe45561095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:32.422570 20779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c741918c906544ca9f766bd0901d8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:32.422878 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c741918c906544ca9f766bd0901d8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:32.424522 20645 tablet_service.cc:1467] Processing CreateTablet for tablet c741918c906544ca9f766bd0901d8985 (DEFAULT_TABLE table=test-table [id=aa5509d7068c4e78ac8254fe45561095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:34:32.425410 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c741918c906544ca9f766bd0901d8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:34:32.428855 20996 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Bootstrap starting.
I20260927 00:34:32.429737 20997 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Bootstrap starting.
I20260927 00:34:32.431386 20996 tablet_bootstrap.cc:654] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:32.431506 20998 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Bootstrap starting.
I20260927 00:34:32.432339 20997 tablet_bootstrap.cc:654] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:32.432833 20996 log.cc:824] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:32.433102 20997 log.cc:824] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:32.433780 20998 tablet_bootstrap.cc:654] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Neither blocks nor log segments found. Creating new log.
I20260927 00:34:32.434554 20998 log.cc:824] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Log is configured to *not* fsync() on all Append() calls
I20260927 00:34:32.434849 20997 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: No bootstrap required, opened a new log
I20260927 00:34:32.434849 20996 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: No bootstrap required, opened a new log
I20260927 00:34:32.435032 20996 ts_tablet_manager.cc:1397] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:34:32.435032 20997 ts_tablet_manager.cc:1397] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:34:32.436136 20998 tablet_bootstrap.cc:492] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: No bootstrap required, opened a new log
I20260927 00:34:32.436335 20998 ts_tablet_manager.cc:1397] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:34:32.438786 20997 raft_consensus.cc:348] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.438786 20996 raft_consensus.cc:348] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.439042 20997 raft_consensus.cc:374] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:32.439042 20996 raft_consensus.cc:374] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:32.439093 20996 raft_consensus.cc:729] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f322ecf8290a41bd867c83125a69e220, State: Initialized, Role: FOLLOWER
I20260927 00:34:32.439093 20997 raft_consensus.cc:729] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fdd498923e49fb976f8668cf002b6f, State: Initialized, Role: FOLLOWER
I20260927 00:34:32.439071 20998 raft_consensus.cc:348] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.439368 20998 raft_consensus.cc:374] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:34:32.439435 20998 raft_consensus.cc:729] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb4097bf6734a6182ce56161e8c6aea, State: Initialized, Role: FOLLOWER
I20260927 00:34:32.439607 20996 consensus_queue.cc:260] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.439607 20997 consensus_queue.cc:260] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.439852 20998 consensus_queue.cc:260] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.440552 20996 ts_tablet_manager.cc:1428] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:34:32.440552 20997 ts_tablet_manager.cc:1428] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:34:32.440810 20998 ts_tablet_manager.cc:1428] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260927 00:34:32.440848 20845 heartbeater.cc:503] Master 127.10.11.254:36343 was elected leader, sending a full tablet report...
I20260927 00:34:32.441253 20979 heartbeater.cc:503] Master 127.10.11.254:36343 was elected leader, sending a full tablet report...
I20260927 00:34:32.442119 20711 heartbeater.cc:503] Master 127.10.11.254:36343 was elected leader, sending a full tablet report...
W20260927 00:34:32.469157 20712 tablet.cc:2369] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:34:32.622771 20980 tablet.cc:2369] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:32.645493 21004 raft_consensus.cc:482] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:34:32.645696 21004 raft_consensus.cc:504] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.647027 21004 leader_election.cc:288] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57fdd498923e49fb976f8668cf002b6f (127.10.11.194:34971), f322ecf8290a41bd867c83125a69e220 (127.10.11.195:39301)
I20260927 00:34:32.650759 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.647340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52133 (local address 127.10.11.195:39301)
0927 00:34:32.647501 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.647506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.647612 (+ 106us) server_negotiation.cc:407] Connection header received
0927 00:34:32.647812 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.647818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.647873 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.647956 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:32.648227 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.648725 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.649762 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.650248 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.650357 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.650410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.650585 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.650646 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:34:32.650777 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.647241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:39301 (local address 127.10.11.193:52133)
0927 00:34:32.647500 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.647516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.647670 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.647977 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.647980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.647997 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:32.648118 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.648123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.648833 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.648838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.649646 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.649654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.650347 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.650350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.650447 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.650475 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.650560 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.650629 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":21,"thread_start_us":88,"threads_started":1}
I20260927 00:34:32.651165 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.647241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:34971 (local address 127.10.11.193:44039)
0927 00:34:32.647939 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.647952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:34:32.648059 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:34:32.648289 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:34:32.648292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:34:32.648306 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:34:32.648438 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.648443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.649129 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.649134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.649904 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:34:32.649912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.650731 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.650735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:34:32.650843 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.650875 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:34:32.650960 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:34:32.651030 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":285,"thread_start_us":122,"threads_started":1}
I20260927 00:34:32.651185 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.647348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44039 (local address 127.10.11.194:34971)
0927 00:34:32.647505 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.647510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.648024 (+ 514us) server_negotiation.cc:407] Connection header received
0927 00:34:32.648138 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.648141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.648192 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.648269 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:34:32.648542 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.649023 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.650025 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.650611 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.650739 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.650784 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.650995 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.651076 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:34:32.651443 20933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c741918c906544ca9f766bd0901d8985" candidate_uuid: "fcb4097bf6734a6182ce56161e8c6aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f322ecf8290a41bd867c83125a69e220" is_pre_election: true
I20260927 00:34:32.651687 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c741918c906544ca9f766bd0901d8985" candidate_uuid: "fcb4097bf6734a6182ce56161e8c6aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fdd498923e49fb976f8668cf002b6f" is_pre_election: true
I20260927 00:34:32.651769 20933 raft_consensus.cc:2454] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb4097bf6734a6182ce56161e8c6aea in term 0.
I20260927 00:34:32.651981 20799 raft_consensus.cc:2454] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb4097bf6734a6182ce56161e8c6aea in term 0.
I20260927 00:34:32.652240 20598 leader_election.cc:302] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f322ecf8290a41bd867c83125a69e220, fcb4097bf6734a6182ce56161e8c6aea; no voters:
I20260927 00:34:32.652516 21004 raft_consensus.cc:2781] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:34:32.652590 21004 raft_consensus.cc:482] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:34:32.652619 21004 raft_consensus.cc:3037] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:32.653610 21004 raft_consensus.cc:504] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.653978 21004 leader_election.cc:288] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [CANDIDATE]: Term 1 election: Requested vote from peers 57fdd498923e49fb976f8668cf002b6f (127.10.11.194:34971), f322ecf8290a41bd867c83125a69e220 (127.10.11.195:39301)
I20260927 00:34:32.654282 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c741918c906544ca9f766bd0901d8985" candidate_uuid: "fcb4097bf6734a6182ce56161e8c6aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fdd498923e49fb976f8668cf002b6f"
I20260927 00:34:32.654331 20933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c741918c906544ca9f766bd0901d8985" candidate_uuid: "fcb4097bf6734a6182ce56161e8c6aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f322ecf8290a41bd867c83125a69e220"
I20260927 00:34:32.654397 20799 raft_consensus.cc:3037] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:32.654460 20933 raft_consensus.cc:3037] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:34:32.655390 20799 raft_consensus.cc:2454] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb4097bf6734a6182ce56161e8c6aea in term 1.
I20260927 00:34:32.655478 20933 raft_consensus.cc:2454] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb4097bf6734a6182ce56161e8c6aea in term 1.
I20260927 00:34:32.655644 20600 leader_election.cc:302] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fdd498923e49fb976f8668cf002b6f, fcb4097bf6734a6182ce56161e8c6aea; no voters:
I20260927 00:34:32.655841 21004 raft_consensus.cc:2781] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:34:32.656064 21004 raft_consensus.cc:686] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 1 LEADER]: Becoming Leader. State: Replica: fcb4097bf6734a6182ce56161e8c6aea, State: Running, Role: LEADER
I20260927 00:34:32.656371 21004 consensus_queue.cc:237] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } }
I20260927 00:34:32.659405 20518 catalog_manager.cc:5495] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea reported cstate change: term changed from 0 to 1, leader changed from <none> to fcb4097bf6734a6182ce56161e8c6aea (127.10.11.193). New cstate: current_term: 1 leader_uuid: "fcb4097bf6734a6182ce56161e8c6aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcb4097bf6734a6182ce56161e8c6aea" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46039 } health_report { overall_health: HEALTHY } } }
W20260927 00:34:32.667783 20846 tablet.cc:2369] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:34:32.706265 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.702339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.10.11.193:46039)
0927 00:34:32.702467 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:34:32.702471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:34:32.702504 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:34:32.702751 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:34:32.702753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:34:32.702811 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:34:32.702932 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:34:32.703206 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.703691 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.704870 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:34:32.705071 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:34:32.705144 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:34:32.705449 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:34:32.705819 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:34:32.705935 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:34:32.706030 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:34:32.706083 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:34:32.712055 20799 raft_consensus.cc:1261] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 1 FOLLOWER]: Refusing update from remote peer fcb4097bf6734a6182ce56161e8c6aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:32.712081 20933 raft_consensus.cc:1261] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 1 FOLLOWER]: Refusing update from remote peer fcb4097bf6734a6182ce56161e8c6aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:34:32.712746 21007 consensus_queue.cc:1035] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fdd498923e49fb976f8668cf002b6f" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:32.712908 21004 consensus_queue.cc:1035] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f322ecf8290a41bd867c83125a69e220" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 39301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:34:32.718643 21010 mvcc.cc:205] Tried to move back new op lower bound from 7333762141022040064 to 7333762140803092480. Current Snapshot: MvccSnapshot[applied={T|T < 7333762141022040064}]
I20260927 00:34:32.719085 21011 mvcc.cc:205] Tried to move back new op lower bound from 7333762141022040064 to 7333762140803092480. Current Snapshot: MvccSnapshot[applied={T|T < 7333762141022040064}]
I20260927 00:34:32.727824 20518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:32.727978 20518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:34:32.730001 20518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b72b4f03bd2a48b5950c9acfe34e181e: Sending DeleteTablet for 3 replicas of tablet c741918c906544ca9f766bd0901d8985
I20260927 00:34:32.730676 20779 tablet_service.cc:1514] Processing DeleteTablet for tablet c741918c906544ca9f766bd0901d8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:32 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:36546
I20260927 00:34:32.730803 20913 tablet_service.cc:1514] Processing DeleteTablet for tablet c741918c906544ca9f766bd0901d8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:32 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:52692
I20260927 00:34:32.730815 20645 tablet_service.cc:1514] Processing DeleteTablet for tablet c741918c906544ca9f766bd0901d8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:34:32 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45704
I20260927 00:34:32.731118 21020 tablet_replica.cc:330] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: stopping tablet replica
I20260927 00:34:32.731161 21021 tablet_replica.cc:330] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: stopping tablet replica
I20260927 00:34:32.731344 21022 tablet_replica.cc:330] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea: stopping tablet replica
I20260927 00:34:32.731454 21021 raft_consensus.cc:2229] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:32.731563 21022 raft_consensus.cc:2229] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:34:32.731566 21020 raft_consensus.cc:2229] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:34:32.731729 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20582
I20260927 00:34:32.731734 21021 raft_consensus.cc:2258] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:32.731855 21022 raft_consensus.cc:2258] T c741918c906544ca9f766bd0901d8985 P fcb4097bf6734a6182ce56161e8c6aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:32.732689 21020 raft_consensus.cc:2258] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:34:32.733794 21021 ts_tablet_manager.cc:1905] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:32.734845 21020 ts_tablet_manager.cc:1905] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:34:32.736366 21021 ts_tablet_manager.cc:1918] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:32.736462 21021 log.cc:1196] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c741918c906544ca9f766bd0901d8985
I20260927 00:34:32.736800 21021 ts_tablet_manager.cc:1939] T c741918c906544ca9f766bd0901d8985 P f322ecf8290a41bd867c83125a69e220: Deleting consensus metadata
I20260927 00:34:32.737485 21020 ts_tablet_manager.cc:1918] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:34:32.737576 21020 log.cc:1196] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c741918c906544ca9f766bd0901d8985
I20260927 00:34:32.737874 21020 ts_tablet_manager.cc:1939] T c741918c906544ca9f766bd0901d8985 P 57fdd498923e49fb976f8668cf002b6f: Deleting consensus metadata
I20260927 00:34:32.738006 20504 catalog_manager.cc:4841] TS f322ecf8290a41bd867c83125a69e220 (127.10.11.195:39301): tablet c741918c906544ca9f766bd0901d8985 (table test-table [id=aa5509d7068c4e78ac8254fe45561095]) successfully deleted
W20260927 00:34:32.738709 20504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:46039 (error 108)
I20260927 00:34:32.739033 20506 catalog_manager.cc:4841] TS 57fdd498923e49fb976f8668cf002b6f (127.10.11.194:34971): tablet c741918c906544ca9f766bd0901d8985 (table test-table [id=aa5509d7068c4e78ac8254fe45561095]) successfully deleted
I20260927 00:34:32.739558 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20716
I20260927 00:34:32.739691 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:34:32.739386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46039 (local address 127.0.0.1:45708)
0927 00:34:32.739535 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:34:32.739601 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 00:34:32.740140 20504 catalog_manager.cc:4584] TS fcb4097bf6734a6182ce56161e8c6aea (127.10.11.193:46039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c741918c906544ca9f766bd0901d8985: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46039: connect: Connection refused (error 111)
I20260927 00:34:32.744879 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20850
I20260927 00:34:32.750319 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 20488
2026-09-27T00:34:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2365 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[21026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[21026](info): set up 2 sockets
Sep 27 00:34:32 dist-test-slave-6gxc krb5kdc[21026](info): commencing operation
krb5kdc: starting...
W20260927 00:34:33.783222 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:34:33 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:34:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangeradmin_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangeradmin_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangerlookup_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangerlookup_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
I20260927 00:34:33.940798 10287 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAm2zct/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/postgres -l logfile start
2026-09-27 00:34:36.356 UTC [21056] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:34:36.356 UTC [21056] LOG: listening on IPv4 address "127.10.11.212", port 39659
2026-09-27 00:34:36.360 UTC [21056] LOG: listening on Unix socket "/tmp/.s.PGSQL.39659"
2026-09-27 00:34:36.394 UTC [21059] LOG: database system was shut down at 2026-09-27 00:34:34 UTC
2026-09-27 00:34:36.398 UTC [21056] LOG: database system is ready to accept connections
I20260927 00:34:37.338869 10287 mini_postgres.cc:96] Postgres bound to 39659
2026-09-27 00:34:37.345 UTC [21067] FATAL: database "slave" does not exist
127.10.11.212:39659 - accepting connections
I20260927 00:34:37.345678 10287 mini_ranger.cc:161] Starting Ranger...
I20260927 00:34:37.360176 10287 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:34:37.819125 10287 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:34:37.819242 10287 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin
2026-09-27 00:34:38,096 [I] DB FLAVOR :POSTGRES
2026-09-27 00:34:38,096 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:34:38,096 [I] Checking connection..
2026-09-27 00:34:38,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:34:38,346 [I] Checking connection passed.
2026-09-27 00:34:38,347 [I] --------- Verifying version history table ---------
2026-09-27 00:34:38,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:34:38,596 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:34:38,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:34:38,832 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:34:38,832 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:34:38,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:34:39.062 UTC [21131] WARNING: there is no transaction in progress
2026-09-27 00:34:39,068 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:34:39,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:34:39,309 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:34:39,309 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:34:39,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:34:39,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:34:39,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6gxc.c.gcp-upstream.internal','N') ;"
2026-09-27 00:34:40,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:34:40,249 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:34:40,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:34:40,468 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:34:40,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:34:40,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:34:40,953 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:34:40,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:34:41.847 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.861 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.876 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.891 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.909 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.928 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.984 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:41.990 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.004 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.015 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.027 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.042 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.052 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.059 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.065 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42.071 UTC [21239] WARNING: there is no transaction in progress
2026-09-27 00:34:42,639 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:34:42,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:34:42,863 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:34:42,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:34:43,104 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:34:43,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:34:43,332 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:34:43,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:34:43,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:34:43,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6gxc.c.gcp-upstream.internal';"
2026-09-27 00:34:44,045 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:34:44,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:34:44,293 [I] DB_PATCHES have already been applied
I20260927 00:34:44.298895 10287 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAm2zct/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:34:44.298975 10287 mini_ranger.cc:193] Using host: 127.10.11.212
I20260927 00:34:44.302213 10287 mini_ranger.cc:227] Ranger admin URL: http://127.10.11.212:59165
Sep 27, 2026 12:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:34:45.395716 10287 mini_ranger.cc:160] Time spent starting Ranger: real 8.050s user 0.000s sys 0.009s
Sep 27, 2026 12:35:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:35:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 12:35:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:35:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:35:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:35:06.428932 10287 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangerkms_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/rangerkms_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.11.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.11.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:35:06 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469306, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:35:06.513900 10287 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:35:06.523227 10287 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:35:07.367933 10287 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:35:07.368029 10287 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms
I20260927 00:35:07.368127 10287 mini_ranger_kms.cc:227] Using postgres at 127.10.11.212:39659
2026-09-27 00:35:07,549 [I] DB FLAVOR :POSTGRES
2026-09-27 00:35:07,549 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:35:07,549 [I] Checking connection
2026-09-27 00:35:07,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:35:07,782 [I] connection success
2026-09-27 00:35:07,782 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:35:07,782 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:35:07,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:35:08,035 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:35:08,036 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:35:08,036 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:35:08,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:39659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:35:08,281 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:35:08.693462 10287 mini_ranger_kms.cc:310] Created kms service
I20260927 00:35:08.882339 10287 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:35:08.926698 10287 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:35:08 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469306, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:09.328303 10287 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:35:09.328466 10287 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:35:09.328514 10287 mini_ranger_kms.cc:240] Using host: 127.10.11.212
I20260927 00:35:09.331966 10287 mini_ranger_kms.cc:276] Ranger KMS PID: 21419
I20260927 00:35:09.332105 10287 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.11.212:59815
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 00:35:11 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:35:11.610968 10287 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.097s user 0.000s sys 0.166s
I20260927 00:35:11.611095 10287 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:35:11.611196 10287 mini_ranger_kms.cc:401] 127.10.11.212:59815/kms/v1/keys
Sep 27 00:35:12 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
Sep 27 00:35:12 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
Sep 27 00:35:14 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:35:14.675215 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34013
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:34013
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:59815/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:35:14.784730 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:35:14.785024 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:35:14.785101 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:35:14.789130 21470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:35:14.789215 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:35:14.789254 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:35:14.789309 21470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:35:14.789345 21470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:35:14.794760 21470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.11.212:59815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:34013
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:35:14.795785 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:35:14.796774 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:35:14.805322 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:35:14.805330 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:14.805444 21470 server_base.cc:1034] running on GCE node
W20260927 00:35:14.805646 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:14.806290 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:35:14.807616 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:35:14.808818 21470 hybrid_clock.cc:648] HybridClock initialized: now 1790469314808783 us; error 46 us; skew 500 ppm
Sep 27 00:35:14 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:35:14.812019 21470 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:35:14.812927 21470 webserver.cc:460] Webserver started at http://127.10.11.254:36899/ using document root <none> and password file <none>
I20260927 00:35:14.813531 21470 fs_manager.cc:359] Metadata directory not provided
I20260927 00:35:14.813584 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:35:14.813793 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:35:14 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:14.873126 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "81973fe41fab4c349819af9b477f9a50"
format_stamp: "Formatted at 2026-09-27 00:35:14 on dist-test-slave-6gxc"
server_key: "b2c64978369b41731b3407c21cb652ca"
server_key_iv: "e69d684ba388409dc6f8be0b6a7ba754"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:14.873754 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "81973fe41fab4c349819af9b477f9a50"
format_stamp: "Formatted at 2026-09-27 00:35:14 on dist-test-slave-6gxc"
server_key: "b2c64978369b41731b3407c21cb652ca"
server_key_iv: "e69d684ba388409dc6f8be0b6a7ba754"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:14.877758 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:35:14.895082 21485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:14.896500 21470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:35:14.896641 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "81973fe41fab4c349819af9b477f9a50"
format_stamp: "Formatted at 2026-09-27 00:35:14 on dist-test-slave-6gxc"
server_key: "b2c64978369b41731b3407c21cb652ca"
server_key_iv: "e69d684ba388409dc6f8be0b6a7ba754"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:14.896775 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:35:14.924176 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:35:14.927336 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:35:14.927547 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:35:14.935531 21470 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:34013
I20260927 00:35:14.935544 21537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:34013 every 8 connection(s)
I20260927 00:35:14.936506 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:35:14.939736 21538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:35:14.942735 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 21470
I20260927 00:35:14.942869 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:35:14.943157 10287 external_mini_cluster.cc:1428] Setting key 98ec63521cb16b59311e2de8369c78e0
I20260927 00:35:14.945776 21538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: Bootstrap starting.
I20260927 00:35:14.948397 21538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: Neither blocks nor log segments found. Creating new log.
I20260927 00:35:14.949242 21538 log.cc:824] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: Log is configured to *not* fsync() on all Append() calls
I20260927 00:35:14.951412 21538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: No bootstrap required, opened a new log
Sep 27 00:35:14 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:35:14.955849 21538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } }
I20260927 00:35:14.956105 21538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:35:14.956199 21538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81973fe41fab4c349819af9b477f9a50, State: Initialized, Role: FOLLOWER
I20260927 00:35:14.956669 21538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } }
I20260927 00:35:14.956821 21538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:35:14.956902 21538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:35:14.957037 21538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:35:14.958171 21538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } }
I20260927 00:35:14.958523 21538 leader_election.cc:302] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81973fe41fab4c349819af9b477f9a50; no voters:
I20260927 00:35:14.958925 21538 leader_election.cc:288] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:35:14.958979 21543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:35:14.959234 21543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [term 1 LEADER]: Becoming Leader. State: Replica: 81973fe41fab4c349819af9b477f9a50, State: Running, Role: LEADER
I20260927 00:35:14.959568 21543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } }
I20260927 00:35:14.959975 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:14.944658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.10.11.254:34013)
0927 00:35:14.945109 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:14.945128 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:14.945170 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:35:14.945964 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:14.945987 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:14.946425 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:14.946769 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:35:14.947632 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:14.949095 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:14.950351 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:14.951024 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:14.951182 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:14.954295 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:35:14.954302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:35:14.954321 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:35:14.954352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:35:14.956410 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:14.956949 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:14.956955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:14.956961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:14.957096 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:14.957406 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:14.957410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:14.957419 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:14.957909 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:35:14.958155 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:14.958784 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 00:35:14.959057 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260927 00:35:14.960213 21538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:35:14.960980 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81973fe41fab4c349819af9b477f9a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } } }
I20260927 00:35:14.961189 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [sys.catalog]: This master's current role is: LEADER
I20260927 00:35:14.961393 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81973fe41fab4c349819af9b477f9a50. Latest consensus state: current_term: 1 leader_uuid: "81973fe41fab4c349819af9b477f9a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81973fe41fab4c349819af9b477f9a50" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 34013 } } }
I20260927 00:35:14.961513 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50 [sys.catalog]: This master's current role is: LEADER
I20260927 00:35:14.962311 21550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:35:14.965163 21550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:35:14.965565 21538 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:35:14.975015 21550 catalog_manager.cc:1347] Generated new cluster ID: 612059ec09c849ca856f5e375b5738e7
I20260927 00:35:14.975086 21550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:35:14.988302 21550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:35:14.989230 21550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:35:15.004897 21550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: Generated new TSK 0
I20260927 00:35:15.005657 21550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:35:15.709770 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:59815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:35:15.852018 21583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:35:15.852242 21583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:35:15.852283 21583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:35:15.858351 21583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:35:15.858428 21583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:35:15.858521 21583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:35:15.866317 21583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.11.212:59815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:35:15.867578 21583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:35:15.868959 21583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:35:15.881570 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:35:15.881564 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:15.882174 21583 server_base.cc:1034] running on GCE node
W20260927 00:35:15.882330 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:15.882781 21583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:35:15.883581 21583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:35:15.884780 21583 hybrid_clock.cc:648] HybridClock initialized: now 1790469315884749 us; error 45 us; skew 500 ppm
Sep 27 00:35:15 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:35:15.888269 21583 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:35:15.889123 21583 webserver.cc:460] Webserver started at http://127.10.11.193:38325/ using document root <none> and password file <none>
I20260927 00:35:15.889684 21583 fs_manager.cc:359] Metadata directory not provided
I20260927 00:35:15.889743 21583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:35:15.889948 21583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:35:15 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:15.907908 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "83179e44be0a496cbea68727de63d65e"
format_stamp: "Formatted at 2026-09-27 00:35:15 on dist-test-slave-6gxc"
server_key: "96be99ea9d48bdd613a5fd09127f61f6"
server_key_iv: "e7eccd5dfc32e4c96f0569a7fda6a235"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:15.908707 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "83179e44be0a496cbea68727de63d65e"
format_stamp: "Formatted at 2026-09-27 00:35:15 on dist-test-slave-6gxc"
server_key: "96be99ea9d48bdd613a5fd09127f61f6"
server_key_iv: "e7eccd5dfc32e4c96f0569a7fda6a235"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:15.915289 21583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 00:35:15.932749 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:15.948647 21583 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.000s
I20260927 00:35:15.948875 21583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "83179e44be0a496cbea68727de63d65e"
format_stamp: "Formatted at 2026-09-27 00:35:15 on dist-test-slave-6gxc"
server_key: "96be99ea9d48bdd613a5fd09127f61f6"
server_key_iv: "e7eccd5dfc32e4c96f0569a7fda6a235"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:15.949169 21583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:35:15.974790 21583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:35:15.977842 21583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:35:15.978086 21583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:35:15.978762 21583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:35:15.979676 21583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:35:15.979746 21583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:15.979820 21583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:35:15.979872 21583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:15.992468 21583 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:46261
I20260927 00:35:15.993665 21583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:35:15.994356 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 21583
I20260927 00:35:15.994494 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:35:15.994817 10287 external_mini_cluster.cc:1428] Setting key bc94b3c0b76297fc398fd72338554bdc
I20260927 00:35:15.996001 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:46261 every 8 connection(s)
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:35:16.014796 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:15.997582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:41207 (local address 127.10.11.254:34013)
0927 00:35:16.002280 (+ 4698us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.002288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.002302 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:35:16.002342 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.002344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.002389 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.003875 (+ 1486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:35:16.003978 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.005428 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.007087 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.007493 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.008285 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.010176 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:35:16.010182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:35:16.010190 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:35:16.010213 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:35:16.011749 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.012252 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.012257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.012259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.012368 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.012722 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.012727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.012729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.012962 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:35:16.013067 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.014506 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.014627 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4623,"thread_start_us":107,"threads_started":1}
I20260927 00:35:16.015870 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:15.998153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34013 (local address 127.10.11.193:41207)
0927 00:35:15.999264 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:15.999320 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.000290 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.002709 (+ 2419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.002719 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.003077 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:35:16.003733 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.003751 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.005555 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.005559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.006931 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.006945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.007698 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.007706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.007928 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.008639 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:35:16.008655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:35:16.009966 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:35:16.011880 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.011886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.011901 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.012152 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.012476 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.012479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.012481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.012627 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.013120 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:35:16.013126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:35:16.013423 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.013626 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.013945 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":113,"threads_started":1}
I20260927 00:35:16.017614 21716 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34013
I20260927 00:35:16.017833 21716 heartbeater.cc:464] Registering TS with master...
I20260927 00:35:16.018388 21716 heartbeater.cc:511] Master 127.10.11.254:34013 requested a full tablet report, sending...
I20260927 00:35:16.020227 21502 ts_manager.cc:194] Registered new tserver with Master: 83179e44be0a496cbea68727de63d65e (127.10.11.193:46261)
I20260927 00:35:16.021730 21502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:41207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:35:16.040709 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:59815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:35:16.189811 21722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:35:16.190227 21722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:35:16.190308 21722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:35:16.196182 21722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:35:16.196300 21722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:35:16.196415 21722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:35:16.203981 21722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.11.212:59815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:35:16.205405 21722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:35:16.206773 21722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:35:16.219813 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:16.220374 21722 server_base.cc:1034] running on GCE node
W20260927 00:35:16.220422 21731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:35:16.220723 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:16.221156 21722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:35:16.221853 21722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:35:16.223039 21722 hybrid_clock.cc:648] HybridClock initialized: now 1790469316223005 us; error 52 us; skew 500 ppm
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:35:16.226320 21722 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:35:16.227312 21722 webserver.cc:460] Webserver started at http://127.10.11.194:33979/ using document root <none> and password file <none>
I20260927 00:35:16.228085 21722 fs_manager.cc:359] Metadata directory not provided
I20260927 00:35:16.228190 21722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:35:16.228499 21722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:16.248700 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "7fb4761d571f4b599c6d72b310b42da7"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "ba65c80ba9b823436bc55bd731f20c69"
server_key_iv: "dd92d763146ea3ebd5562a31d843266c"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.249629 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "7fb4761d571f4b599c6d72b310b42da7"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "ba65c80ba9b823436bc55bd731f20c69"
server_key_iv: "dd92d763146ea3ebd5562a31d843266c"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.254973 21722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:35:16.276871 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.278549 21722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:35:16.278729 21722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "7fb4761d571f4b599c6d72b310b42da7"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "ba65c80ba9b823436bc55bd731f20c69"
server_key_iv: "dd92d763146ea3ebd5562a31d843266c"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.278940 21722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:35:16.309782 21722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:35:16.312896 21722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:35:16.313095 21722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:35:16.313750 21722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:35:16.314653 21722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:35:16.314724 21722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.314832 21722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:35:16.314898 21722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.335875 21722 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:33725
I20260927 00:35:16.335965 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:33725 every 8 connection(s)
I20260927 00:35:16.337134 21722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:35:16.342387 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 21722
I20260927 00:35:16.342540 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:35:16.342831 10287 external_mini_cluster.cc:1428] Setting key 904fe2218392096941ef71fd1bd82643
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:35:16.354350 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.340878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54765 (local address 127.10.11.254:34013)
0927 00:35:16.341012 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.341017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.341424 (+ 407us) server_negotiation.cc:407] Connection header received
0927 00:35:16.342676 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.342680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.342732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.342787 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:35:16.344320 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.345157 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.346851 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.347185 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.347261 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.350276 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:35:16.350280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:35:16.350283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:35:16.350309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:35:16.352050 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.352654 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.352658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.352660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.352739 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.353128 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.353131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.353133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.353338 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:35:16.353418 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.354042 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.354188 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:35:16.356211 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.339807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34013 (local address 127.10.11.194:54765)
0927 00:35:16.341268 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.341321 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.342415 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.343176 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.343184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.343527 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:35:16.344154 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.344171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.345310 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.345313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.346721 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.346730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.347456 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.347460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.347683 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.348351 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:35:16.348372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:35:16.350059 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:35:16.352188 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.352193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.352208 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.352537 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.352843 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.352846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.352848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.353017 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.353482 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:35:16.353500 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:35:16.353779 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.354730 (+ 951us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.355065 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":132,"threads_started":1}
I20260927 00:35:16.357692 21852 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34013
I20260927 00:35:16.357980 21852 heartbeater.cc:464] Registering TS with master...
I20260927 00:35:16.358577 21852 heartbeater.cc:511] Master 127.10.11.254:34013 requested a full tablet report, sending...
I20260927 00:35:16.359719 21502 ts_manager.cc:194] Registered new tserver with Master: 7fb4761d571f4b599c6d72b310b42da7 (127.10.11.194:33725)
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:35:16.360451 21502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:54765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:35:16.380736 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:59815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
W20260927 00:35:16.543718 21857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:35:16.544116 21857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:35:16.544210 21857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:35:16.550091 21857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:35:16.550248 21857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:35:16.550374 21857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:35:16.557637 21857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.11.212:59815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:35:16.559705 21857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:35:16.561077 21857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:35:16.572697 21863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:35:16.572726 21862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:35:16.573601 21865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:35:16.574599 21857 server_base.cc:1034] running on GCE node
I20260927 00:35:16.575001 21857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:35:16.575676 21857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:35:16.576849 21857 hybrid_clock.cc:648] HybridClock initialized: now 1790469316576790 us; error 72 us; skew 500 ppm
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:35:16.580224 21857 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:35:16.581286 21857 webserver.cc:460] Webserver started at http://127.10.11.195:42047/ using document root <none> and password file <none>
I20260927 00:35:16.581998 21857 fs_manager.cc:359] Metadata directory not provided
I20260927 00:35:16.582087 21857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:35:16.582348 21857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:16.599110 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "e35de504028544adbebc828ce49b715e"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "1da790abcc3e68e0a4c76734a8b1790f"
server_key_iv: "72d925ffe28699bd354b76a0e2280eea"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.599977 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "e35de504028544adbebc828ce49b715e"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "1da790abcc3e68e0a4c76734a8b1790f"
server_key_iv: "72d925ffe28699bd354b76a0e2280eea"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.610399 21857 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260927 00:35:16.629067 21872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.630800 21857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 00:35:16.630923 21857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "e35de504028544adbebc828ce49b715e"
format_stamp: "Formatted at 2026-09-27 00:35:16 on dist-test-slave-6gxc"
server_key: "1da790abcc3e68e0a4c76734a8b1790f"
server_key_iv: "72d925ffe28699bd354b76a0e2280eea"
server_key_version: "kuduclusterkey@0"
I20260927 00:35:16.631064 21857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:35:16.647897 21857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:35:16.651796 21857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:35:16.652065 21857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:35:16.652813 21857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:35:16.653918 21857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:35:16.654038 21857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.654170 21857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:35:16.654243 21857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:35:16.664940 21857 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:42045
I20260927 00:35:16.666179 21857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:35:16.672613 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:42045 every 8 connection(s)
I20260927 00:35:16.675268 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 21857
I20260927 00:35:16.675391 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:35:16.675685 10287 external_mini_cluster.cc:1428] Setting key 378dba81e61442ca8eed4d1e829b5325
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:35:16.691658 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.675333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:48319 (local address 127.10.11.254:34013)
0927 00:35:16.675473 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.675478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.678607 (+ 3129us) server_negotiation.cc:407] Connection header received
0927 00:35:16.679749 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.679752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.679803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.679897 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:35:16.681336 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.682204 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.683896 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.684272 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.684368 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.687323 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:35:16.687327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:35:16.687330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:35:16.687357 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:35:16.689316 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.689880 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.689884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.689886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.689977 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.690387 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.690390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.690392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.690608 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:35:16.690705 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.691321 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.691451 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:35:16.692528 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.675836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:34013 (local address 127.10.11.195:48319)
0927 00:35:16.678424 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.678485 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.679517 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.680118 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.680127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.680485 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:35:16.681175 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.681196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.682352 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.682356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.683757 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.683767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.684427 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.684431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.684619 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.685400 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:35:16.685420 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:35:16.687147 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:35:16.689443 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.689449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.689466 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.689791 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.690085 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:35:16.690088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:35:16.690090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:35:16.690288 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:35:16.690719 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:35:16.690725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:35:16.691035 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.691228 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.691539 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2354,"thread_start_us":156,"threads_started":1}
I20260927 00:35:16.693629 21986 heartbeater.cc:347] Connected to a master server at 127.10.11.254:34013
I20260927 00:35:16.693877 21986 heartbeater.cc:464] Registering TS with master...
I20260927 00:35:16.694492 21986 heartbeater.cc:511] Master 127.10.11.254:34013 requested a full tablet report, sending...
I20260927 00:35:16.695576 21502 ts_manager.cc:194] Registered new tserver with Master: e35de504028544adbebc828ce49b715e (127.10.11.195:42045)
I20260927 00:35:16.696281 21502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:48319
I20260927 00:35:16.701566 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:35:16.713346 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.704070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.10.11.254:34013)
0927 00:35:16.704179 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.704183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.704470 (+ 287us) server_negotiation.cc:407] Connection header received
0927 00:35:16.704597 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.704600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.704644 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.704727 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:35:16.705601 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.706466 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.707622 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.707939 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.708042 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.709762 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:35:16.709766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:35:16.709769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:35:16.709794 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:35:16.711568 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.712066 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.712070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.712072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.712155 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:35:16.712468 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:35:16.712471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:35:16.712473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:35:16.712676 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:35:16.712804 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.713079 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.713191 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:35:16.716837 21502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:35:16.721068 21502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:35:16.739653 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.732499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42045 (local address 127.0.0.1:45996)
0927 00:35:16.733663 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.733674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.733735 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.734810 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.734813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.734830 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:35:16.734925 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.734931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.736519 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.736523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.738022 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.738032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.739157 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.739160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.739265 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.739308 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.739391 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.739500 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":71,"threads_started":1}
I20260927 00:35:16.740638 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.731449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:33725 (local address 127.0.0.1:38270)
0927 00:35:16.733063 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.733114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.733232 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.734378 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.734385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.734411 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:35:16.734557 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.734565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.737000 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.737004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.738336 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.738347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.740272 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.740276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.740366 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.740390 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.740444 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.740510 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":100,"threads_started":1}
I20260927 00:35:16.740666 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.732957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.10.11.195:42045)
0927 00:35:16.734340 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.734351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.734372 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:35:16.734441 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.734446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.734617 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.735073 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:35:16.735197 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.736391 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.738210 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.739025 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.740244 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.740394 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.740443 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.740535 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":139,"threads_started":1}
I20260927 00:35:16.741532 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.731605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.10.11.194:33725)
0927 00:35:16.733802 (+ 2197us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.733818 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.733843 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:35:16.733923 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.733929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.734188 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.734728 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:35:16.734854 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.736846 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.738508 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.740129 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.741094 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.741234 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.741290 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.741388 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":143,"threads_started":1}
I20260927 00:35:16.743855 21920 tablet_service.cc:1467] Processing CreateTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 (DEFAULT_TABLE table=test-table [id=53c821888ce34ed2a737634ef4855c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:35:16.744426 21785 tablet_service.cc:1467] Processing CreateTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 (DEFAULT_TABLE table=test-table [id=53c821888ce34ed2a737634ef4855c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:35:16.745007 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:35:16.745425 21785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:35:16.751768 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.732206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46261 (local address 127.0.0.1:45842)
0927 00:35:16.733438 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.733450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.733519 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.745692 (+ 12173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.745696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.745715 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:35:16.745810 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.745816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.748910 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.748913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.750088 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.750121 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.751356 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.751359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.751447 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.751472 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.751538 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.751618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":84,"threads_started":1}
I20260927 00:35:16.754925 22005 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Bootstrap starting.
I20260927 00:35:16.754992 22004 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Bootstrap starting.
I20260927 00:35:16.756230 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.732369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.10.11.193:46261)
0927 00:35:16.745207 (+ 12838us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.745215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.745240 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:35:16.745317 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.745324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.745526 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.745962 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:35:16.746119 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.747160 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.750260 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.751044 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.755754 (+ 4710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.755932 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.755995 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.756093 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12719,"thread_start_us":106,"threads_started":1}
I20260927 00:35:16.758661 22005 tablet_bootstrap.cc:654] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Neither blocks nor log segments found. Creating new log.
I20260927 00:35:16.759145 21644 tablet_service.cc:1467] Processing CreateTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 (DEFAULT_TABLE table=test-table [id=53c821888ce34ed2a737634ef4855c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:35:16.759588 22005 log.cc:824] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:35:16.760265 21644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:35:16.769433 22004 tablet_bootstrap.cc:654] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Neither blocks nor log segments found. Creating new log.
I20260927 00:35:16.770438 22004 log.cc:824] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:35:16.770516 22007 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: Bootstrap starting.
I20260927 00:35:16.772601 22004 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: No bootstrap required, opened a new log
I20260927 00:35:16.772785 22007 tablet_bootstrap.cc:654] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: Neither blocks nor log segments found. Creating new log.
I20260927 00:35:16.772797 22004 ts_tablet_manager.cc:1397] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260927 00:35:16.773456 22007 log.cc:824] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:35:16.775399 22007 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: No bootstrap required, opened a new log
I20260927 00:35:16.775401 22005 tablet_bootstrap.cc:492] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: No bootstrap required, opened a new log
I20260927 00:35:16.775594 22005 ts_tablet_manager.cc:1397] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20260927 00:35:16.775630 22007 ts_tablet_manager.cc:1397] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:35:16.776738 22004 raft_consensus.cc:348] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.777010 22004 raft_consensus.cc:374] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:35:16.777056 22004 raft_consensus.cc:729] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35de504028544adbebc828ce49b715e, State: Initialized, Role: FOLLOWER
I20260927 00:35:16.777552 22004 consensus_queue.cc:260] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.778555 22004 ts_tablet_manager.cc:1428] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:35:16.778894 21986 heartbeater.cc:503] Master 127.10.11.254:34013 was elected leader, sending a full tablet report...
I20260927 00:35:16.780365 22005 raft_consensus.cc:348] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.780365 22007 raft_consensus.cc:348] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.780627 22007 raft_consensus.cc:374] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:35:16.780704 22005 raft_consensus.cc:374] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:35:16.780725 22007 raft_consensus.cc:729] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83179e44be0a496cbea68727de63d65e, State: Initialized, Role: FOLLOWER
I20260927 00:35:16.780803 22005 raft_consensus.cc:729] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb4761d571f4b599c6d72b310b42da7, State: Initialized, Role: FOLLOWER
I20260927 00:35:16.781266 22005 consensus_queue.cc:260] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.781266 22007 consensus_queue.cc:260] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.782296 22005 ts_tablet_manager.cc:1428] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 00:35:16.782657 21852 heartbeater.cc:503] Master 127.10.11.254:34013 was elected leader, sending a full tablet report...
I20260927 00:35:16.783154 22007 ts_tablet_manager.cc:1428] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260927 00:35:16.783479 21716 heartbeater.cc:503] Master 127.10.11.254:34013 was elected leader, sending a full tablet report...
Sep 27 00:35:16 dist-test-slave-6gxc krb5kdc[21026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:16.827812 22011 raft_consensus.cc:482] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:35:16.828063 22011 raft_consensus.cc:504] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.829456 22011 leader_election.cc:288] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83179e44be0a496cbea68727de63d65e (127.10.11.193:46261), 7fb4761d571f4b599c6d72b310b42da7 (127.10.11.194:33725)
I20260927 00:35:16.834862 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.829672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46261 (local address 127.10.11.195:50617)
0927 00:35:16.830008 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.830026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.830296 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.830954 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.830958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.830977 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:35:16.831094 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.831101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.832158 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.832162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.833420 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.833431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.834434 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.834438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.834558 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.834582 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.834656 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.834727 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260927 00:35:16.835922 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.830212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:50617 (local address 127.10.11.193:46261)
0927 00:35:16.830382 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.830387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.830402 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:35:16.830456 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.830459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.830516 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.830589 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:35:16.831232 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.832020 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.833577 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.834210 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.834325 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.834377 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.835724 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.835803 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 00:35:16.836534 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b0c4db5d5e4e04b9f8b6d82eb842d1" candidate_uuid: "e35de504028544adbebc828ce49b715e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83179e44be0a496cbea68727de63d65e" is_pre_election: true
I20260927 00:35:16.836848 21668 raft_consensus.cc:2454] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35de504028544adbebc828ce49b715e in term 0.
I20260927 00:35:16.837417 21873 leader_election.cc:302] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83179e44be0a496cbea68727de63d65e, e35de504028544adbebc828ce49b715e; no voters:
I20260927 00:35:16.838757 22011 raft_consensus.cc:2781] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:35:16.838850 22011 raft_consensus.cc:482] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:35:16.838891 22011 raft_consensus.cc:3037] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:35:16.840204 22011 raft_consensus.cc:504] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
W20260927 00:35:16.840816 21853 tablet.cc:2369] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:35:16.841019 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b0c4db5d5e4e04b9f8b6d82eb842d1" candidate_uuid: "e35de504028544adbebc828ce49b715e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83179e44be0a496cbea68727de63d65e"
I20260927 00:35:16.841140 21668 raft_consensus.cc:3037] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:35:16.842545 21668 raft_consensus.cc:2454] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35de504028544adbebc828ce49b715e in term 1.
I20260927 00:35:16.842931 21873 leader_election.cc:302] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83179e44be0a496cbea68727de63d65e, e35de504028544adbebc828ce49b715e; no voters:
I20260927 00:35:16.844058 22015 raft_consensus.cc:2781] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:35:16.844324 22015 raft_consensus.cc:686] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 1 LEADER]: Becoming Leader. State: Replica: e35de504028544adbebc828ce49b715e, State: Running, Role: LEADER
I20260927 00:35:16.844756 22011 leader_election.cc:288] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [CANDIDATE]: Term 1 election: Requested vote from peers 83179e44be0a496cbea68727de63d65e (127.10.11.193:46261), 7fb4761d571f4b599c6d72b310b42da7 (127.10.11.194:33725)
I20260927 00:35:16.844758 22015 consensus_queue.cc:237] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } }
I20260927 00:35:16.846141 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.829939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:33725 (local address 127.10.11.195:47627)
0927 00:35:16.835025 (+ 5086us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.835039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:35:16.835117 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:35:16.837456 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:35:16.837459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:35:16.837491 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:35:16.837581 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.837591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.843286 (+ 5695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.843290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.844476 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:35:16.844496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.845714 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.845718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:35:16.845808 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.845829 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:35:16.845893 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:35:16.845965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5031,"thread_start_us":159,"threads_started":1}
I20260927 00:35:16.846576 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.830772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:47627 (local address 127.10.11.194:33725)
0927 00:35:16.830897 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.830901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.835294 (+ 4393us) server_negotiation.cc:407] Connection header received
0927 00:35:16.835351 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.835354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.835401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.835468 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:35:16.837724 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.838529 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.844830 (+ 6301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.845457 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.845568 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.845621 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.846361 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.846444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:35:16.847532 21795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b0c4db5d5e4e04b9f8b6d82eb842d1" candidate_uuid: "e35de504028544adbebc828ce49b715e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb4761d571f4b599c6d72b310b42da7" is_pre_election: true
I20260927 00:35:16.847843 21795 raft_consensus.cc:2454] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35de504028544adbebc828ce49b715e in term 0.
I20260927 00:35:16.848138 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b0c4db5d5e4e04b9f8b6d82eb842d1" candidate_uuid: "e35de504028544adbebc828ce49b715e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb4761d571f4b599c6d72b310b42da7"
I20260927 00:35:16.848232 21806 raft_consensus.cc:3037] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:35:16.849148 21502 catalog_manager.cc:5495] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e reported cstate change: term changed from 0 to 1, leader changed from <none> to e35de504028544adbebc828ce49b715e (127.10.11.195). New cstate: current_term: 1 leader_uuid: "e35de504028544adbebc828ce49b715e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35de504028544adbebc828ce49b715e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 } health_report { overall_health: UNKNOWN } } }
I20260927 00:35:16.849498 21806 raft_consensus.cc:2454] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35de504028544adbebc828ce49b715e in term 1.
I20260927 00:35:16.864454 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.859416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.10.11.195:42045)
0927 00:35:16.859542 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:35:16.859546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:35:16.859562 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:35:16.859751 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:35:16.859755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:35:16.859817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:35:16.859907 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:35:16.860271 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.861045 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.862306 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:35:16.862659 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:35:16.862761 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:35:16.863164 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:35:16.863707 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:35:16.863822 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:35:16.864231 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 00:35:16.864292 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:35:16.871986 21806 raft_consensus.cc:1261] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 1 FOLLOWER]: Refusing update from remote peer e35de504028544adbebc828ce49b715e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:35:16.873016 22017 consensus_queue.cc:1035] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb4761d571f4b599c6d72b310b42da7" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:35:16.873176 21668 raft_consensus.cc:1261] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 83179e44be0a496cbea68727de63d65e [term 1 FOLLOWER]: Refusing update from remote peer e35de504028544adbebc828ce49b715e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:35:16.873766 22017 consensus_queue.cc:1035] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83179e44be0a496cbea68727de63d65e" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:35:16.887856 22020 mvcc.cc:205] Tried to move back new op lower bound from 7333762321899454464 to 7333762321800302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333762321899454464}]
I20260927 00:35:16.890435 22021 mvcc.cc:205] Tried to move back new op lower bound from 7333762321899454464 to 7333762321800302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333762321899454464}]
I20260927 00:35:16.895355 22022 mvcc.cc:205] Tried to move back new op lower bound from 7333762321899454464 to 7333762321800302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333762321899454464}]
I20260927 00:35:16.900799 21502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:35:16.900960 21502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:35:16.903952 21502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81973fe41fab4c349819af9b477f9a50: Sending DeleteTablet for 3 replicas of tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1
I20260927 00:35:16.905179 21644 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:35:16 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45842
I20260927 00:35:16.905416 21920 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:35:16 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45996
I20260927 00:35:16.905798 21785 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:35:16 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:38270
I20260927 00:35:16.906415 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 21583
I20260927 00:35:16.910288 22030 tablet_replica.cc:330] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: stopping tablet replica
I20260927 00:35:16.910518 22031 tablet_replica.cc:330] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: stopping tablet replica
I20260927 00:35:16.910920 22031 raft_consensus.cc:2229] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:35:16.911242 22031 raft_consensus.cc:2258] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:35:16.911696 22030 raft_consensus.cc:2229] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:35:16.912815 22031 ts_tablet_manager.cc:1905] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:35:16.913309 22030 raft_consensus.cc:2258] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:35:16.916323 22031 ts_tablet_manager.cc:1918] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:35:16.916488 22031 log.cc:1196] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/01b0c4db5d5e4e04b9f8b6d82eb842d1
I20260927 00:35:16.916890 22031 ts_tablet_manager.cc:1939] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P 7fb4761d571f4b599c6d72b310b42da7: Deleting consensus metadata
I20260927 00:35:16.918232 21489 catalog_manager.cc:4841] TS 7fb4761d571f4b599c6d72b310b42da7 (127.10.11.194:33725): tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 (table test-table [id=53c821888ce34ed2a737634ef4855c35]) successfully deleted
I20260927 00:35:16.919382 22030 ts_tablet_manager.cc:1905] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:35:16.921322 22030 ts_tablet_manager.cc:1918] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:35:16.921454 22030 log.cc:1196] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/01b0c4db5d5e4e04b9f8b6d82eb842d1
I20260927 00:35:16.921789 22030 ts_tablet_manager.cc:1939] T 01b0c4db5d5e4e04b9f8b6d82eb842d1 P e35de504028544adbebc828ce49b715e: Deleting consensus metadata
I20260927 00:35:16.925489 21487 catalog_manager.cc:4841] TS e35de504028544adbebc828ce49b715e (127.10.11.195:42045): tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1 (table test-table [id=53c821888ce34ed2a737634ef4855c35]) successfully deleted
W20260927 00:35:16.926250 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:46261 (error 108)
I20260927 00:35:16.926327 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 21722
I20260927 00:35:16.934440 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 21857
I20260927 00:35:16.936964 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:35:16.936607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46261 (local address 127.0.0.1:45850)
0927 00:35:16.936802 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 00:35:16.936877 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260927 00:35:16.937175 21486 catalog_manager.cc:4584] TS 83179e44be0a496cbea68727de63d65e (127.10.11.193:46261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01b0c4db5d5e4e04b9f8b6d82eb842d1: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46261: connect: Connection refused (error 111)
I20260927 00:35:16.942008 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 21470
I20260927 00:35:16.948681 10287 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:35:17.293510 10287 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:35:17.293656 10287 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:35:17.638206 10287 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:35:17.638 UTC [21056] LOG: received smart shutdown request
2026-09-27 00:35:17.641 UTC [21056] LOG: background worker "logical replication launcher" (PID 21065) exited with exit code 1
2026-09-27 00:35:17.641 UTC [21060] LOG: shutting down
2026-09-27 00:35:17.798 UTC [21056] LOG: database system is shut down
2026-09-27T00:35:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45158 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:35:17 dist-test-slave-6gxc krb5kdc[22041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:35:17 dist-test-slave-6gxc krb5kdc[22041](info): set up 2 sockets
Sep 27 00:35:17 dist-test-slave-6gxc krb5kdc[22041](info): commencing operation
krb5kdc: starting...
W20260927 00:35:19.006146 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:35:19 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:35:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangeradmin_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangeradmin_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangerlookup_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangerlookup_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
I20260927 00:35:19.165112 10287 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAm2zct/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/postgres -l logfile start
2026-09-27 00:35:21.539 UTC [22071] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:35:21.539 UTC [22071] LOG: listening on IPv4 address "127.10.11.212", port 42469
2026-09-27 00:35:21.542 UTC [22071] LOG: listening on Unix socket "/tmp/.s.PGSQL.42469"
2026-09-27 00:35:21.576 UTC [22074] LOG: database system was shut down at 2026-09-27 00:35:20 UTC
2026-09-27 00:35:21.579 UTC [22071] LOG: database system is ready to accept connections
I20260927 00:35:22.525938 10287 mini_postgres.cc:96] Postgres bound to 42469
2026-09-27 00:35:22.532 UTC [22082] FATAL: database "slave" does not exist
127.10.11.212:42469 - accepting connections
I20260927 00:35:22.532600 10287 mini_ranger.cc:161] Starting Ranger...
I20260927 00:35:22.546682 10287 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:35:22.995733 10287 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:35:22.995818 10287 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin
2026-09-27 00:35:23,268 [I] DB FLAVOR :POSTGRES
2026-09-27 00:35:23,268 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:35:23,269 [I] Checking connection..
2026-09-27 00:35:23,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:35:23,518 [I] Checking connection passed.
2026-09-27 00:35:23,518 [I] --------- Verifying version history table ---------
2026-09-27 00:35:23,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:35:23,761 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:35:23,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:35:24,018 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:35:24,019 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:35:24,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:35:24.271 UTC [22146] WARNING: there is no transaction in progress
2026-09-27 00:35:24,277 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:35:24,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:35:24,506 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:35:24,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:35:24,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:35:24,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:35:25,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6gxc.c.gcp-upstream.internal','N') ;"
2026-09-27 00:35:25,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:35:25,505 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:35:25,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:35:25,756 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:35:25,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:35:25,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:35:26,232 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:35:26,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:35:27.176 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.191 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.207 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.223 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.239 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.255 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.307 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.313 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.327 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.337 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.347 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.360 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.369 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.375 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.382 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27.388 UTC [22254] WARNING: there is no transaction in progress
2026-09-27 00:35:27,998 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:35:27,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:35:28,230 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:35:28,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:35:28,489 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:35:28,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:35:28,742 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:35:28,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:35:28,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:35:29,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6gxc.c.gcp-upstream.internal';"
2026-09-27 00:35:29,414 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:35:29,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:35:29,661 [I] DB_PATCHES have already been applied
I20260927 00:35:29.667234 10287 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAm2zct/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:35:29.667292 10287 mini_ranger.cc:193] Using host: 127.10.11.212
I20260927 00:35:29.670590 10287 mini_ranger.cc:227] Ranger admin URL: http://127.10.11.212:53823
Sep 27, 2026 12:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:35:30.772217 10287 mini_ranger.cc:160] Time spent starting Ranger: real 8.240s user 0.006s sys 0.003s
Sep 27, 2026 12:35:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:35:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:35:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:35:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:35:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:35:52.245267 10287 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.11.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.11.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangerkms_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.11.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/rangerkms_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.11.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.11.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.11.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/HTTP_127.10.11.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:35:52 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:35:52.327961 10287 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:35:52.337330 10287 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:35:53.106961 10287 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:35:53.107054 10287 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms
I20260927 00:35:53.107074 10287 mini_ranger_kms.cc:227] Using postgres at 127.10.11.212:42469
2026-09-27 00:35:53,279 [I] DB FLAVOR :POSTGRES
2026-09-27 00:35:53,279 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:35:53,279 [I] Checking connection
2026-09-27 00:35:53,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:35:53,515 [I] connection success
2026-09-27 00:35:53,515 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:35:53,516 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:35:53,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:35:53,752 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:35:53,752 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:35:53,752 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:35:53,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.11.212:42469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:35:54,025 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:35:54.498538 10287 mini_ranger_kms.cc:310] Created kms service
I20260927 00:35:54.676410 10287 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:35:54.718467 10287 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:35:54 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:35:55.101109 10287 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:35:55.101264 10287 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAm2zct/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAm2zct/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:35:55.101286 10287 mini_ranger_kms.cc:240] Using host: 127.10.11.212
I20260927 00:35:55.105183 10287 mini_ranger_kms.cc:276] Ranger KMS PID: 22434
I20260927 00:35:55.105265 10287 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.11.212:48227
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:35:57.361730 10287 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.034s user 0.000s sys 0.156s
I20260927 00:35:57.361855 10287 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:35:57.361935 10287 mini_ranger_kms.cc:401] 127.10.11.212:48227/kms/v1/keys
Sep 27 00:35:57 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469357, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:35:57 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469357, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
Sep 27 00:35:58 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469357, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.11.212@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
Sep 27 00:36:00 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:36:00.340961 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45167
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:45167
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:00.449891 22485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:00.450172 22485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:00.450243 22485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:00.454208 22485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:00.454254 22485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:00.454274 22485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:00.454311 22485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:00.454340 22485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:00.459583 22485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.11.212:48227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:45167
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:00.460467 22485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:00.461396 22485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:00.469446 22490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:00.469532 22491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:00.469434 22493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:00.469681 22485 server_base.cc:1034] running on GCE node
I20260927 00:36:00.470271 22485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:00.471237 22485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:00.472426 22485 hybrid_clock.cc:648] HybridClock initialized: now 1790469360472408 us; error 28 us; skew 500 ppm
Sep 27 00:36:00 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469360, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:00.479568 22485 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:00.480379 22485 webserver.cc:460] Webserver started at http://127.10.11.254:43885/ using document root <none> and password file <none>
I20260927 00:36:00.480880 22485 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:00.480978 22485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:00.481240 22485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:36:00 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469360, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:00.526726 22485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "2886bf3ffc90477b89ff4edd37b2dd8b"
format_stamp: "Formatted at 2026-09-27 00:36:00 on dist-test-slave-6gxc"
server_key: "7c71709a6f3a6488341953f9ca72808e"
server_key_iv: "84a3641c4780c89c754267545213ae9d"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:00.527334 22485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "2886bf3ffc90477b89ff4edd37b2dd8b"
format_stamp: "Formatted at 2026-09-27 00:36:00 on dist-test-slave-6gxc"
server_key: "7c71709a6f3a6488341953f9ca72808e"
server_key_iv: "84a3641c4780c89c754267545213ae9d"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:00.531235 22485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:36:00.547523 22500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:00.548888 22485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:36:00.549031 22485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2886bf3ffc90477b89ff4edd37b2dd8b"
format_stamp: "Formatted at 2026-09-27 00:36:00 on dist-test-slave-6gxc"
server_key: "7c71709a6f3a6488341953f9ca72808e"
server_key_iv: "84a3641c4780c89c754267545213ae9d"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:00.549163 22485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:00.565040 22485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:00.567516 22485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:00.567716 22485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:00.574790 22485 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:45167
I20260927 00:36:00.574818 22552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:45167 every 8 connection(s)
I20260927 00:36:00.575707 22485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:00.577270 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 22485
I20260927 00:36:00.577427 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:36:00.577728 10287 external_mini_cluster.cc:1428] Setting key 565b5ab045104ea21e3379d3e058aaa4
I20260927 00:36:00.578523 22553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:00.584026 22553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Bootstrap starting.
Sep 27 00:36:00 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:00.586664 22553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:00.587352 22553 log.cc:824] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:00.589533 22553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: No bootstrap required, opened a new log
I20260927 00:36:00.592360 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:00.579113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.10.11.254:45167)
0927 00:36:00.579523 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:00.579538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:00.579570 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:36:00.580223 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:00.580239 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:00.580538 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:00.580860 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:00.581683 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:00.582728 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:00.583595 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:00.584045 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:00.584227 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:00.587004 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:00.587009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:00.587021 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:00.587046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:00.589462 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:00.589928 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:00.589934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:00.589939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:00.590064 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:00.590345 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:00.590348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:00.590350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:00.590818 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:00.591000 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:00.591208 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:36:00.591468 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":140,"threads_started":1}
I20260927 00:36:00.593340 22553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:00.593612 22553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:00.593672 22553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886bf3ffc90477b89ff4edd37b2dd8b, State: Initialized, Role: FOLLOWER
I20260927 00:36:00.594163 22553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:00.594324 22553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:00.594430 22553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:00.594533 22553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:00.595353 22553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:00.595636 22553 leader_election.cc:302] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886bf3ffc90477b89ff4edd37b2dd8b; no voters:
I20260927 00:36:00.595873 22553 leader_election.cc:288] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:36:00.595993 22558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:00.596244 22558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 LEADER]: Becoming Leader. State: Replica: 2886bf3ffc90477b89ff4edd37b2dd8b, State: Running, Role: LEADER
I20260927 00:36:00.596550 22558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:00.597019 22553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:00.598181 22559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } } }
I20260927 00:36:00.598299 22559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:00.598572 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886bf3ffc90477b89ff4edd37b2dd8b. Latest consensus state: current_term: 1 leader_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } } }
I20260927 00:36:00.598644 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:00.598691 22567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:00.601431 22567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:00.601825 22553 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:36:00.609835 22567 catalog_manager.cc:1347] Generated new cluster ID: f2b27385f7784a438070c405f71c9b1e
I20260927 00:36:00.609907 22567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:00.635854 22567 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:36:00.637073 22567 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:00.647002 22567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Generated new TSK 0
I20260927 00:36:00.647765 22567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:36:01.293015 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:01.436192 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:01.436554 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:01.436617 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:01.442677 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:01.442757 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:01.442845 22598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:01.450678 22598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:01.452015 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:01.453374 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:01.463094 22606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:01.463593 22605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:01.463094 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:01.463898 22598 server_base.cc:1034] running on GCE node
I20260927 00:36:01.464306 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:01.464999 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:01.466177 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790469361466129 us; error 88 us; skew 500 ppm
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:01.469163 22598 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:01.470162 22598 webserver.cc:460] Webserver started at http://127.10.11.193:43273/ using document root <none> and password file <none>
I20260927 00:36:01.470714 22598 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:01.470810 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:01.471040 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:01.499794 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "d33991514c16433486a30cd5939686e4"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "c15825bf222370d24321ecf1eb6f1ddb"
server_key_iv: "63cf505fa7cb3a50c3db2bc57b79db6f"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.500568 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "d33991514c16433486a30cd5939686e4"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "c15825bf222370d24321ecf1eb6f1ddb"
server_key_iv: "63cf505fa7cb3a50c3db2bc57b79db6f"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.505087 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:36:01.521868 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.523347 22598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:36:01.523485 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "d33991514c16433486a30cd5939686e4"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "c15825bf222370d24321ecf1eb6f1ddb"
server_key_iv: "63cf505fa7cb3a50c3db2bc57b79db6f"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.523622 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:01.548643 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:01.552269 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:01.552506 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:01.553253 22598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:01.554287 22598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:01.554389 22598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.554513 22598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:01.554593 22598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.564750 22598 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44149
I20260927 00:36:01.565977 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:01.570791 22730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44149 every 8 connection(s)
I20260927 00:36:01.571527 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 22598
I20260927 00:36:01.571668 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:36:01.572005 10287 external_mini_cluster.cc:1428] Setting key eb720f9508095af8690bc6dbc14537f1
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:01.598770 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:01.571067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:48933 (local address 127.10.11.254:45167)
0927 00:36:01.571503 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:01.571513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:01.572110 (+ 597us) server_negotiation.cc:407] Connection header received
0927 00:36:01.573355 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:01.573358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:01.573411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:01.573494 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:01.588274 (+ 14780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.589118 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.590917 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.591260 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.591330 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:01.594455 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:01.594459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:01.594462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:01.594489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:01.596300 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:01.596920 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:01.596923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:01.596925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:01.597015 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:01.597451 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:01.597455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:01.597457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:01.597674 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:01.597761 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:01.598464 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 00:36:01.598589 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":94,"threads_started":1}
I20260927 00:36:01.600553 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:01.569950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.193:48933)
0927 00:36:01.571885 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:01.571942 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:01.573029 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:01.573781 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:01.573794 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:01.574178 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:01.588047 (+ 13869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:01.588072 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.589318 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.589322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:01.590776 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:01.590788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.591541 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.591544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:01.591764 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:01.592485 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:01.592506 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:01.594267 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:01.596432 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:01.596438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:01.596453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:01.596802 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:01.597117 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:01.597120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:01.597122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:01.597340 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:01.597829 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:01.597835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:01.598157 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 00:36:01.599250 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0927 00:36:01.599572 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"thread_start_us":174,"threads_started":1}
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:01.604303 22731 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:01.604542 22731 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:01.605075 22731 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:01.606747 22517 ts_manager.cc:194] Registered new tserver with Master: d33991514c16433486a30cd5939686e4 (127.10.11.193:44149)
I20260927 00:36:01.608278 22517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:48933
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:36:01.647274 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:01.797847 22737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:01.798282 22737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:01.798398 22737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:01.804396 22737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:01.804534 22737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:01.804661 22737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:01.812309 22737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:01.813807 22737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:01.815196 22737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:01.827518 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:01.828969 22737 server_base.cc:1034] running on GCE node
W20260927 00:36:01.835078 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:01.835289 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:01.835903 22737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:01.836689 22737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:01.837865 22737 hybrid_clock.cc:648] HybridClock initialized: now 1790469361837802 us; error 78 us; skew 500 ppm
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:01.841635 22737 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:01.842705 22737 webserver.cc:460] Webserver started at http://127.10.11.194:44501/ using document root <none> and password file <none>
I20260927 00:36:01.843386 22737 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:01.843454 22737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:01.843662 22737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:01.861595 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "e873cd962ef640799a3df8a504b019cc"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "6725d88222ef5b302448a4e2ad061dae"
server_key_iv: "f152773fe936e9962bdbb7b9f732a5af"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.862473 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "e873cd962ef640799a3df8a504b019cc"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "6725d88222ef5b302448a4e2ad061dae"
server_key_iv: "f152773fe936e9962bdbb7b9f732a5af"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.867760 22737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:36:01.893244 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.896217 22737 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260927 00:36:01.896344 22737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "e873cd962ef640799a3df8a504b019cc"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "6725d88222ef5b302448a4e2ad061dae"
server_key_iv: "f152773fe936e9962bdbb7b9f732a5af"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:01.896462 22737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:01.949227 22737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:01.952837 22737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:01.953066 22737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:01.953799 22737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:01.954875 22737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:01.954979 22737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.955082 22737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:01.955149 22737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:01.968200 22737 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:37153
I20260927 00:36:01.969338 22737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:01.969632 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 22737
I20260927 00:36:01.969764 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:36:01.970062 10287 external_mini_cluster.cc:1428] Setting key 4d0ff2a808c5711a0e628ec8872c3784
I20260927 00:36:01.976212 22866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:37153 every 8 connection(s)
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Sep 27 00:36:01 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:01.998204 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:01.978274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:55999 (local address 127.10.11.254:45167)
0927 00:36:01.978393 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:01.978398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:01.980266 (+ 1868us) server_negotiation.cc:407] Connection header received
0927 00:36:01.981450 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:01.981454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:01.981505 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:01.981556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:01.983267 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.984156 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.990015 (+ 5859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.990401 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.990482 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:01.993550 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:01.993554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:01.993557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:01.993586 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:01.995334 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:01.995950 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:01.995953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:01.995955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:01.996036 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:01.996416 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:01.996420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:01.996421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:01.996630 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:01.996706 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:01.997849 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0927 00:36:01.997984 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:36:02.002898 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:01.978628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.194:55999)
0927 00:36:01.980115 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:01.980169 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:01.981208 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:01.982068 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:01.982077 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:01.982439 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:01.983099 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:01.983118 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.984313 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.984316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:01.985695 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:01.985706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:01.990694 (+ 4988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:01.990697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:01.990914 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:01.991599 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:01.991620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:01.993356 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:01.995471 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:01.995477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:01.995496 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:01.995835 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:01.996135 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:01.996138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:01.996140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:01.996313 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:01.996782 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:01.996789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:01.997099 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:36:01.999076 (+ 1977us) client_negotiation.cc:240] Negotiation successful
0927 00:36:01.999425 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":129,"threads_started":1}
I20260927 00:36:02.004424 22867 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:02.004668 22867 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:02.005228 22867 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:02.006361 22517 ts_manager.cc:194] Registered new tserver with Master: e873cd962ef640799a3df8a504b019cc (127.10.11.194:37153)
I20260927 00:36:02.007079 22517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:55999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:36:02.030761 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
W20260927 00:36:02.195641 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:02.195988 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:02.196070 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:02.202462 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:02.202548 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:02.202647 22872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:02.210204 22872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:02.211441 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:02.212713 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:02.223985 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:02.223985 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:02.224239 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:02.224447 22872 server_base.cc:1034] running on GCE node
I20260927 00:36:02.224810 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:02.225430 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:02.226660 22872 hybrid_clock.cc:648] HybridClock initialized: now 1790469362226638 us; error 35 us; skew 500 ppm
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:02.229465 22872 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:02.230410 22872 webserver.cc:460] Webserver started at http://127.10.11.195:43411/ using document root <none> and password file <none>
I20260927 00:36:02.230944 22872 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:02.231002 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:02.231208 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:02.248376 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "d5a6b5c6f9354387bcc73154cca5b81b"
format_stamp: "Formatted at 2026-09-27 00:36:02 on dist-test-slave-6gxc"
server_key: "967108056fbbd74bfc07e4fac10e56c1"
server_key_iv: "57d0f77d575a94bd1534596ceca5d3e3"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:02.249372 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "d5a6b5c6f9354387bcc73154cca5b81b"
format_stamp: "Formatted at 2026-09-27 00:36:02 on dist-test-slave-6gxc"
server_key: "967108056fbbd74bfc07e4fac10e56c1"
server_key_iv: "57d0f77d575a94bd1534596ceca5d3e3"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:02.259486 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260927 00:36:02.281286 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.283612 22872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:36:02.283738 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "d5a6b5c6f9354387bcc73154cca5b81b"
format_stamp: "Formatted at 2026-09-27 00:36:02 on dist-test-slave-6gxc"
server_key: "967108056fbbd74bfc07e4fac10e56c1"
server_key_iv: "57d0f77d575a94bd1534596ceca5d3e3"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:02.283864 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:02.297578 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:02.300968 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:02.301151 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:02.301810 22872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:02.302807 22872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:02.302870 22872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.302928 22872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:02.302963 22872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.314196 22872 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:36701
I20260927 00:36:02.314234 23000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:36701 every 8 connection(s)
I20260927 00:36:02.315320 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:02.319365 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 22872
I20260927 00:36:02.319487 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:36:02.320088 10287 external_mini_cluster.cc:1428] Setting key bc5b222f4591fd61d62dced0eb247ceb
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:02.335312 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:02.317670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:46815 (local address 127.10.11.254:45167)
0927 00:36:02.317821 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:02.317827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:02.318921 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 00:36:02.322906 (+ 3985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:02.322912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:02.322966 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:02.323068 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:02.324443 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.325260 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.326979 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.327320 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.327435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:02.330395 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:02.330399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:02.330402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:02.330435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:02.332190 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:02.332790 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:02.332794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:02.332795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:02.332883 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:02.333486 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:02.333489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:02.333491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:02.334297 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:02.334410 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:02.335014 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 00:36:02.335148 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:36:02.336480 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:02.318222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.195:46815)
0927 00:36:02.318759 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:02.318816 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:02.322670 (+ 3854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:02.323276 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:02.323285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:02.323622 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:02.324283 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:02.324303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.325398 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.325401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:02.326780 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:02.326791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.327479 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.327482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:02.327689 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:02.328465 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:02.328485 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:02.330214 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:02.332315 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:02.332321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:02.332337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:02.332660 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:02.332987 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:02.332990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:02.332992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:02.333179 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:02.334424 (+ 1245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:02.334430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:02.334724 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 00:36:02.334928 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 00:36:02.335235 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260927 00:36:02.338141 23001 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:02.338371 23001 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:02.338878 23001 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:02.341574 22517 ts_manager.cc:194] Registered new tserver with Master: d5a6b5c6f9354387bcc73154cca5b81b (127.10.11.195:36701)
I20260927 00:36:02.342330 22517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:46815
I20260927 00:36:02.345638 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:02.346486 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 22598
I20260927 00:36:02.353403 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 22737
I20260927 00:36:02.361531 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 22872
I20260927 00:36:02.369685 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 22485
I20260927 00:36:02.374856 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45167
--webserver_interface=127.10.11.254
--webserver_port=43885
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:45167
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:02.491240 23004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:02.491466 23004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:02.491504 23004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:02.495383 23004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:02.495429 23004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:02.495448 23004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:02.495464 23004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:02.495481 23004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:02.500511 23004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:45167
--ranger_config_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=43885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:02.501350 23004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:02.502285 23004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:02.510274 23010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:02.510258 23012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:02.510305 23004 server_base.cc:1034] running on GCE node
W20260927 00:36:02.510242 23009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:02.511009 23004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:02.512010 23004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:02.513201 23004 hybrid_clock.cc:648] HybridClock initialized: now 1790469362513182 us; error 30 us; skew 500 ppm
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:02.515820 23004 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:02.516722 23004 webserver.cc:460] Webserver started at http://127.10.11.254:43885/ using document root <none> and password file <none>
I20260927 00:36:02.517231 23004 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:02.517325 23004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:02.579782 23004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:36:02.582031 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.583499 23004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:02.583640 23004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2886bf3ffc90477b89ff4edd37b2dd8b"
format_stamp: "Formatted at 2026-09-27 00:36:00 on dist-test-slave-6gxc"
server_key: "7c71709a6f3a6488341953f9ca72808e"
server_key_iv: "84a3641c4780c89c754267545213ae9d"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:02.584187 23004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:02.596385 23004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:02.599080 23004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:02.599295 23004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:02.606195 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:45167 every 8 connection(s)
I20260927 00:36:02.606221 23004 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:45167
I20260927 00:36:02.607211 23004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:02.611253 23073 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:36:02.611325 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23004
I20260927 00:36:02.611799 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:44149
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:45167
--webserver_port=43273
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:02.613462 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Bootstrap starting.
I20260927 00:36:02.629395 23073 log.cc:824] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:02.635599 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:02.635989 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Bootstrap complete.
I20260927 00:36:02.639972 23073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:02.640322 23073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886bf3ffc90477b89ff4edd37b2dd8b, State: Initialized, Role: FOLLOWER
I20260927 00:36:02.640790 23073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:02.640899 23073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:02.640980 23073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:02.641076 23073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:36:02.642879 23073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:02.643195 23073 leader_election.cc:302] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886bf3ffc90477b89ff4edd37b2dd8b; no voters:
I20260927 00:36:02.643476 23073 leader_election.cc:288] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:36:02.643597 23077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:36:02.643849 23077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [term 2 LEADER]: Becoming Leader. State: Replica: 2886bf3ffc90477b89ff4edd37b2dd8b, State: Running, Role: LEADER
I20260927 00:36:02.644141 23077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } }
I20260927 00:36:02.644639 23073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:02.645473 23078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } } }
I20260927 00:36:02.645601 23078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:02.645841 23079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886bf3ffc90477b89ff4edd37b2dd8b. Latest consensus state: current_term: 2 leader_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886bf3ffc90477b89ff4edd37b2dd8b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45167 } } }
I20260927 00:36:02.645920 23079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:02.646595 23085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:02.649550 23085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:02.650033 23073 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:36:02.650195 23085 catalog_manager.cc:1259] Loaded cluster ID: f2b27385f7784a438070c405f71c9b1e
I20260927 00:36:02.650241 23085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:02.652503 23085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:02.653118 23085 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Loaded TSK: 0
I20260927 00:36:02.654291 23085 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:36:02.749246 23075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:02.749588 23075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:02.749675 23075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:02.756198 23075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:02.756278 23075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:02.756363 23075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:02.763940 23075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:44149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=43273
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:02.765020 23075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:02.766353 23075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:02.776968 23111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:02.776950 23110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:02.777534 23113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:02.777801 23075 server_base.cc:1034] running on GCE node
I20260927 00:36:02.778282 23075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:02.778901 23075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:02.780076 23075 hybrid_clock.cc:648] HybridClock initialized: now 1790469362780046 us; error 40 us; skew 500 ppm
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:02.782799 23075 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:02.783864 23075 webserver.cc:460] Webserver started at http://127.10.11.193:43273/ using document root <none> and password file <none>
I20260927 00:36:02.784510 23075 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:02.784591 23075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:02.819697 23075 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.000s
I20260927 00:36:02.822763 23121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.824414 23075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:36:02.824517 23075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "d33991514c16433486a30cd5939686e4"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "c15825bf222370d24321ecf1eb6f1ddb"
server_key_iv: "63cf505fa7cb3a50c3db2bc57b79db6f"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:02.825160 23075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:02.846823 23075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:02.850452 23075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:02.850654 23075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:02.851353 23075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:02.852331 23075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:02.852383 23075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.852438 23075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:02.852460 23075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:02.864547 23075 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44149
I20260927 00:36:02.864718 23234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44149 every 8 connection(s)
I20260927 00:36:02.865808 23075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:02.869455 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23075
I20260927 00:36:02.869910 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:37153
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:45167
--webserver_port=44501
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:36:02 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:02.896795 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:02.868381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.193:46275)
0927 00:36:02.869153 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:02.871581 (+ 2428us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:02.872622 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:02.874369 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:02.874378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:02.874725 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:02.875373 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:02.875394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.885588 (+ 10194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.885592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:02.887067 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:02.887080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.888253 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.888257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:02.888486 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:02.889175 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:02.889199 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:02.890817 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:02.893397 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:02.893404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:02.893419 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:02.893756 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:02.894233 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:02.894237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:02.894239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:02.894408 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:02.895188 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:02.895195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:02.895481 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 00:36:02.895678 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:36:02.896000 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":140,"threads_started":1}
I20260927 00:36:02.904728 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:02.868232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:46275 (local address 127.10.11.254:45167)
0927 00:36:02.868869 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:02.868892 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:02.872949 (+ 4057us) server_negotiation.cc:407] Connection header received
0927 00:36:02.873271 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:02.873286 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:02.873707 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:02.874064 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:02.875573 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.885427 (+ 9854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.887276 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:02.888042 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:02.888935 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:02.891042 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:02.891048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:02.891066 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:02.891098 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:02.893262 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:02.893934 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:02.893940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:02.893945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:02.894083 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:02.894527 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:02.894530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:02.894532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:02.895034 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:02.903244 (+ 8210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:02.903424 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 00:36:02.903742 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":113,"threads_started":1}
I20260927 00:36:02.907584 23235 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:02.907847 23235 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:02.908386 23235 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:02.910619 23235 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:45167 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:36:02.910733 23235 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:02.910904 23235 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:02.911949 23235 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:02.912099 23235 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:02.912853 23235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:36:03.015969 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:03.016260 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:03.016330 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:03.021044 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:03.021126 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:03.021236 23239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:03.028599 23239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:37153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=44501
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:03.029831 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:03.031136 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:03.042055 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:03.042063 23244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:03.042328 23239 server_base.cc:1034] running on GCE node
W20260927 00:36:03.042078 23245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:03.042857 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:03.043567 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:03.044747 23239 hybrid_clock.cc:648] HybridClock initialized: now 1790469363044725 us; error 40 us; skew 500 ppm
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:03.048403 23239 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:03.049422 23239 webserver.cc:460] Webserver started at http://127.10.11.194:44501/ using document root <none> and password file <none>
I20260927 00:36:03.049923 23239 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:03.049989 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:03.072559 23239 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260927 00:36:03.075882 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.077517 23239 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:36:03.077648 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "e873cd962ef640799a3df8a504b019cc"
format_stamp: "Formatted at 2026-09-27 00:36:01 on dist-test-slave-6gxc"
server_key: "6725d88222ef5b302448a4e2ad061dae"
server_key_iv: "f152773fe936e9962bdbb7b9f732a5af"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:03.078402 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:03.112383 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:03.115780 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:03.115983 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:03.116715 23239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:03.117739 23239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:03.117823 23239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.117877 23239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:03.117908 23239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.130461 23239 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:37153
I20260927 00:36:03.130764 23368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:37153 every 8 connection(s)
I20260927 00:36:03.131716 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:03.132216 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23239
I20260927 00:36:03.132726 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.11.212:48227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:36701
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:45167
--webserver_port=43411
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:03.161406 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:03.143783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.194:40999)
0927 00:36:03.144422 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:03.144488 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:03.145702 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:03.146551 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:03.146561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:03.146927 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:03.147617 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:03.147638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.148788 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.148791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:03.149938 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:03.149948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.151599 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.151605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:03.151811 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:03.152508 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:03.152528 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:03.154272 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:03.158174 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:03.158181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:03.158197 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:03.158553 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:03.158891 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:03.158894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:03.158896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:03.159071 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:03.159561 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:03.159567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:03.159887 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 00:36:03.160081 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 00:36:03.160396 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":135,"threads_started":1}
I20260927 00:36:03.162163 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:03.143266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40999 (local address 127.10.11.254:45167)
0927 00:36:03.143415 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:03.143420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:03.144595 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 00:36:03.145955 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:03.145959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:03.146021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:03.146082 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:03.147797 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.148623 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.150071 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.150430 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.150541 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:03.154584 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:03.154588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:03.154591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:03.154619 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:03.158011 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.158675 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.158678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.158680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.158781 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.159185 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.159189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.159191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.159407 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:03.159506 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:03.161835 (+ 2329us) server_negotiation.cc:299] Negotiation successful
0927 00:36:03.161976 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:36:03.163829 23370 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:03.164053 23370 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.164564 23370 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:03.166126 23370 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:45167 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:36:03.166247 23370 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.166407 23370 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:03.167250 23370 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.167414 23370 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:03.168232 23370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:36:03.266965 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:03.267198 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:03.267237 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:03.271605 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:03.271659 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:03.271714 23369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:03.277885 23369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.11.212:48227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:36701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=43411
--tserver_master_addrs=127.10.11.254:45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:03.279105 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:03.280164 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:03.291746 23379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:03.291759 23381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:03.292218 23369 server_base.cc:1034] running on GCE node
W20260927 00:36:03.294261 23378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:03.294692 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:03.295327 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:03.296504 23369 hybrid_clock.cc:648] HybridClock initialized: now 1790469363296473 us; error 43 us; skew 500 ppm
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:03.299742 23369 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:03.300812 23369 webserver.cc:460] Webserver started at http://127.10.11.195:43411/ using document root <none> and password file <none>
I20260927 00:36:03.301481 23369 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:03.301586 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:03.325449 23369 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 00:36:03.328908 23389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.330672 23369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:36:03.330853 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "d5a6b5c6f9354387bcc73154cca5b81b"
format_stamp: "Formatted at 2026-09-27 00:36:02 on dist-test-slave-6gxc"
server_key: "967108056fbbd74bfc07e4fac10e56c1"
server_key_iv: "57d0f77d575a94bd1534596ceca5d3e3"
server_key_version: "kuduclusterkey@0"
I20260927 00:36:03.331653 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:03.350404 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:03.353840 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:03.354022 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:03.354729 23369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:03.355738 23369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:03.355804 23369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.355859 23369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:03.355892 23369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:03.366930 23369 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:36701
I20260927 00:36:03.368307 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:03.369033 23502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:36701 every 8 connection(s)
I20260927 00:36:03.375407 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23369
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:03.394490 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:03.372159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:39391 (local address 127.10.11.254:45167)
0927 00:36:03.372305 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:03.372309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:03.372328 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:36:03.373278 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:03.373282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:03.373341 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:03.373437 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:03.383319 (+ 9882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.384262 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.386382 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.386825 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.386931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:03.390263 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:03.390267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:03.390270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:03.390297 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:03.391953 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.392564 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.392580 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.392582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.392686 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.393076 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.393080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.393082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.393284 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:03.393362 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:03.394180 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 00:36:03.394307 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":68}
I20260927 00:36:03.396126 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:03.371340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45167 (local address 127.10.11.195:39391)
0927 00:36:03.371891 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:03.371951 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:03.372986 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:03.373658 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:03.373668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:03.376064 (+ 2396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:03.383101 (+ 7037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:03.383130 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.384432 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.384436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:03.386226 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:03.386240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.387128 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.387133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:03.387401 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:03.388207 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:03.388228 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:03.390038 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:03.392086 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:03.392091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:03.392107 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:03.392445 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:03.392794 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:03.392797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:03.392799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:03.392971 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:03.393480 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:03.393487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:03.393831 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 00:36:03.394728 (+ 897us) client_negotiation.cc:240] Negotiation successful
0927 00:36:03.395066 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":126,"threads_started":1}
I20260927 00:36:03.396212 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:03.376526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.10.11.254:45167)
0927 00:36:03.376744 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:03.376753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:03.384399 (+ 7646us) server_negotiation.cc:407] Connection header received
0927 00:36:03.384435 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:03.384438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:03.384493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:03.384578 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:03.386523 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.387493 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.389252 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:03.389848 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:03.389950 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:03.392143 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:03.392147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:03.392149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:03.392173 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:03.394170 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.394889 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.394893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.394895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.394991 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:03.395312 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:03.395320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:03.395322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:03.395496 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:03.395694 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:03.395899 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 00:36:03.396017 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260927 00:36:03.397666 23503 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45167
I20260927 00:36:03.397907 23503 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.398490 23503 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:03.399971 23503 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:45167 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:36:03.400116 23503 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.400311 23503 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:03.401089 23503 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.401295 23503 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
W20260927 00:36:03.401985 23503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:36:03 dist-test-slave-6gxc krb5kdc[22041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:03.913084 23235 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:03.913407 23235 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:03.914629 23036 ts_manager.cc:194] Registered new tserver with Master: d33991514c16433486a30cd5939686e4 (127.10.11.193:44149)
I20260927 00:36:03.916324 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:46275
Sep 27 00:36:04 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:04.170240 23370 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:04.170521 23370 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:04.171279 23036 ts_manager.cc:194] Registered new tserver with Master: e873cd962ef640799a3df8a504b019cc (127.10.11.194:37153)
I20260927 00:36:04.172003 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:40999
Sep 27 00:36:04 dist-test-slave-6gxc krb5kdc[22041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for HTTP/127.10.11.212@KRBTEST.COM
I20260927 00:36:04.402210 23503 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:04.402511 23503 heartbeater.cc:511] Master 127.10.11.254:45167 requested a full tablet report, sending...
I20260927 00:36:04.403323 23036 ts_manager.cc:194] Registered new tserver with Master: d5a6b5c6f9354387bcc73154cca5b81b (127.10.11.195:36701)
I20260927 00:36:04.404107 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:39391
I20260927 00:36:04.414677 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:04.428747 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.416600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.10.11.254:45167)
0927 00:36:04.418348 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.418353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.418368 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:36:04.418409 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.418415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.418471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.418554 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:04.419499 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.420328 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.421636 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.422053 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.422240 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.424190 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:04.424194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:04.424197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:04.424226 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:04.426312 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:04.426861 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:04.426866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:04.426868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:04.426962 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:04.427295 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:04.427299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:04.427301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:04.427529 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:04.427674 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.428346 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.428540 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":92,"threads_started":1}
I20260927 00:36:04.433121 23036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:36:04.437836 23036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:36:04.462004 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.451159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:36701 (local address 127.0.0.1:59244)
0927 00:36:04.452900 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.452911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:04.452980 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:04.453665 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:04.453671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:04.453692 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:04.453811 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.453817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.455305 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.455308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.457715 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.457725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.461491 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.461494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.461596 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.461641 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:36:04.461740 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:36:04.461850 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1692,"thread_start_us":80,"threads_started":1}
I20260927 00:36:04.462657 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.451512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.10.11.195:36701)
0927 00:36:04.452532 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.452539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.453064 (+ 525us) server_negotiation.cc:407] Connection header received
0927 00:36:04.453156 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.453162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.453365 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.453504 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:04.453938 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.455176 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.460369 (+ 5193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.461203 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.461301 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.461436 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.462397 (+ 961us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.462506 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":87,"threads_started":1}
I20260927 00:36:04.463470 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.450750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.10.11.194:37153)
0927 00:36:04.458573 (+ 7823us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.458580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.458612 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:36:04.458677 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.458682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.458839 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.458968 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:04.459326 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.460418 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.461720 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.462497 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.462605 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.462746 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.463257 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.463346 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7718,"thread_start_us":105,"threads_started":1}
I20260927 00:36:04.463171 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.450576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:37153 (local address 127.0.0.1:53500)
0927 00:36:04.452657 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.452670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:04.452747 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:04.459109 (+ 6362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:04.459112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:04.459128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:04.459208 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.459213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.460528 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.460530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.461581 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.461600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.462810 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.462814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.462900 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.462921 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:36:04.462989 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:36:04.463050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"thread_start_us":82,"threads_started":1}
I20260927 00:36:04.465489 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.449019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44149 (local address 127.0.0.1:54598)
0927 00:36:04.452083 (+ 3064us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.452143 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:04.452263 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:04.456850 (+ 4587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:04.456853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:04.456870 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:04.456951 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.456956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.459050 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.459053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.460238 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.460248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.465136 (+ 4888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.465139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.465222 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.465244 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:36:04.465312 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:36:04.465370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2899,"thread_start_us":102,"threads_started":1}
I20260927 00:36:04.466118 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.449846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.10.11.193:44149)
0927 00:36:04.456278 (+ 6432us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.456285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.456307 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:36:04.456376 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.456381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.456557 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.456692 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:04.457858 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.458934 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.462908 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.464849 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.464945 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.465065 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.465870 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.465962 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6326,"thread_start_us":101,"threads_started":1}
I20260927 00:36:04.466892 23437 tablet_service.cc:1467] Processing CreateTablet for tablet bc67cc90f961459fa0ecf068374545c1 (DEFAULT_TABLE table=test-table [id=b012035df37347f3b3f296c82a7c5b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:04.466907 23303 tablet_service.cc:1467] Processing CreateTablet for tablet bc67cc90f961459fa0ecf068374545c1 (DEFAULT_TABLE table=test-table [id=b012035df37347f3b3f296c82a7c5b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:04.467990 23437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc67cc90f961459fa0ecf068374545c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:04.467999 23303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc67cc90f961459fa0ecf068374545c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:04.468647 23169 tablet_service.cc:1467] Processing CreateTablet for tablet bc67cc90f961459fa0ecf068374545c1 (DEFAULT_TABLE table=test-table [id=b012035df37347f3b3f296c82a7c5b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:04.469545 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc67cc90f961459fa0ecf068374545c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:04.477216 23539 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Bootstrap starting.
I20260927 00:36:04.479748 23539 tablet_bootstrap.cc:654] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:04.480444 23539 log.cc:824] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:04.482596 23539 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: No bootstrap required, opened a new log
I20260927 00:36:04.482820 23539 ts_tablet_manager.cc:1397] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:36:04.486399 23541 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Bootstrap starting.
I20260927 00:36:04.486490 23539 raft_consensus.cc:348] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.486794 23539 raft_consensus.cc:374] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:04.486915 23539 raft_consensus.cc:729] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33991514c16433486a30cd5939686e4, State: Initialized, Role: FOLLOWER
I20260927 00:36:04.487748 23539 consensus_queue.cc:260] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.488866 23539 ts_tablet_manager.cc:1428] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:36:04.489243 23235 heartbeater.cc:503] Master 127.10.11.254:45167 was elected leader, sending a full tablet report...
I20260927 00:36:04.490291 23541 tablet_bootstrap.cc:654] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:04.492216 23541 log.cc:824] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:04.495702 23541 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: No bootstrap required, opened a new log
I20260927 00:36:04.495914 23541 ts_tablet_manager.cc:1397] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 00:36:04.495931 23538 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Bootstrap starting.
I20260927 00:36:04.498261 23538 tablet_bootstrap.cc:654] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:04.499480 23538 log.cc:824] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:04.500357 23541 raft_consensus.cc:348] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.500659 23541 raft_consensus.cc:374] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:04.500720 23541 raft_consensus.cc:729] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e873cd962ef640799a3df8a504b019cc, State: Initialized, Role: FOLLOWER
I20260927 00:36:04.501183 23541 consensus_queue.cc:260] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.502682 23541 ts_tablet_manager.cc:1428] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:36:04.503064 23370 heartbeater.cc:503] Master 127.10.11.254:45167 was elected leader, sending a full tablet report...
I20260927 00:36:04.509459 23538 tablet_bootstrap.cc:492] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: No bootstrap required, opened a new log
I20260927 00:36:04.509725 23538 ts_tablet_manager.cc:1397] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260927 00:36:04.513725 23538 raft_consensus.cc:348] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.514005 23538 raft_consensus.cc:374] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:04.514050 23538 raft_consensus.cc:729] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a6b5c6f9354387bcc73154cca5b81b, State: Initialized, Role: FOLLOWER
I20260927 00:36:04.514647 23538 consensus_queue.cc:260] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.516299 23538 ts_tablet_manager.cc:1428] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:36:04.516686 23503 heartbeater.cc:503] Master 127.10.11.254:45167 was elected leader, sending a full tablet report...
I20260927 00:36:04.581156 23545 raft_consensus.cc:482] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:04.581395 23545 raft_consensus.cc:504] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.583042 23545 leader_election.cc:288] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d33991514c16433486a30cd5939686e4 (127.10.11.193:44149), d5a6b5c6f9354387bcc73154cca5b81b (127.10.11.195:36701)
I20260927 00:36:04.593524 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.584393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:56741 (local address 127.10.11.193:44149)
0927 00:36:04.584684 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.584947 (+ 263us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.585279 (+ 332us) server_negotiation.cc:407] Connection header received
0927 00:36:04.585591 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.585595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.585682 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.585747 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:04.586466 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.587335 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.592143 (+ 4808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.592762 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.592872 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.592926 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.593301 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.593388 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260927 00:36:04.594249 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.583426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44149 (local address 127.10.11.194:56741)
0927 00:36:04.585171 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.585196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:04.585398 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:04.585905 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:04.585908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:04.585926 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:04.586073 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.586081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.587480 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.587483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.591964 (+ 4481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.591976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.593025 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.593029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.593147 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.593171 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:36:04.593953 (+ 782us) client_negotiation.cc:240] Negotiation successful
0927 00:36:04.594035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"thread_start_us":125,"threads_started":1}
I20260927 00:36:04.595738 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.584529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:38533 (local address 127.10.11.195:36701)
0927 00:36:04.584817 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.584822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.586389 (+ 1567us) server_negotiation.cc:407] Connection header received
0927 00:36:04.587922 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.587926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.587978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.588029 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:04.588940 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.589686 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.594535 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.595173 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.595277 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.595325 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.595538 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.595630 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260927 00:36:04.595834 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc67cc90f961459fa0ecf068374545c1" candidate_uuid: "e873cd962ef640799a3df8a504b019cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33991514c16433486a30cd5939686e4" is_pre_election: true
I20260927 00:36:04.596071 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.583887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:36701 (local address 127.10.11.194:38533)
0927 00:36:04.586311 (+ 2424us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.586324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:04.587756 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:04.588160 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:04.588163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:04.588178 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:04.588794 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.588809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.589831 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.589834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.591066 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:04.591077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.595302 (+ 4225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.595305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:04.595401 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.595423 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:36:04.595878 (+ 455us) client_negotiation.cc:240] Negotiation successful
0927 00:36:04.595950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2359,"thread_start_us":87,"threads_started":1}
I20260927 00:36:04.596239 23189 raft_consensus.cc:2454] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e873cd962ef640799a3df8a504b019cc in term 0.
I20260927 00:36:04.596735 23257 leader_election.cc:302] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d33991514c16433486a30cd5939686e4, e873cd962ef640799a3df8a504b019cc; no voters:
I20260927 00:36:04.596733 23439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc67cc90f961459fa0ecf068374545c1" candidate_uuid: "e873cd962ef640799a3df8a504b019cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" is_pre_election: true
I20260927 00:36:04.596978 23439 raft_consensus.cc:2454] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e873cd962ef640799a3df8a504b019cc in term 0.
I20260927 00:36:04.597033 23545 raft_consensus.cc:2781] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:04.597143 23545 raft_consensus.cc:482] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:04.597195 23545 raft_consensus.cc:3037] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:04.598582 23545 raft_consensus.cc:504] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.598960 23545 leader_election.cc:288] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [CANDIDATE]: Term 1 election: Requested vote from peers d33991514c16433486a30cd5939686e4 (127.10.11.193:44149), d5a6b5c6f9354387bcc73154cca5b81b (127.10.11.195:36701)
I20260927 00:36:04.599699 23439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc67cc90f961459fa0ecf068374545c1" candidate_uuid: "e873cd962ef640799a3df8a504b019cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a6b5c6f9354387bcc73154cca5b81b"
I20260927 00:36:04.599813 23439 raft_consensus.cc:3037] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:04.601032 23439 raft_consensus.cc:2454] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e873cd962ef640799a3df8a504b019cc in term 1.
I20260927 00:36:04.601420 23256 leader_election.cc:302] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5a6b5c6f9354387bcc73154cca5b81b, e873cd962ef640799a3df8a504b019cc; no voters:
I20260927 00:36:04.601711 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc67cc90f961459fa0ecf068374545c1" candidate_uuid: "e873cd962ef640799a3df8a504b019cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33991514c16433486a30cd5939686e4"
I20260927 00:36:04.601835 23189 raft_consensus.cc:3037] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:04.602775 23545 raft_consensus.cc:2781] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:04.602993 23545 raft_consensus.cc:686] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 1 LEADER]: Becoming Leader. State: Replica: e873cd962ef640799a3df8a504b019cc, State: Running, Role: LEADER
I20260927 00:36:04.603137 23189 raft_consensus.cc:2454] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e873cd962ef640799a3df8a504b019cc in term 1.
I20260927 00:36:04.603348 23545 consensus_queue.cc:237] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } }
I20260927 00:36:04.606729 23036 catalog_manager.cc:5495] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc reported cstate change: term changed from 0 to 1, leader changed from <none> to e873cd962ef640799a3df8a504b019cc (127.10.11.194). New cstate: current_term: 1 leader_uuid: "e873cd962ef640799a3df8a504b019cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e873cd962ef640799a3df8a504b019cc" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 } health_report { overall_health: UNKNOWN } } }
W20260927 00:36:04.618932 23236 tablet.cc:2369] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:04.621697 23504 tablet.cc:2369] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:04.644162 23371 tablet.cc:2369] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:04.652325 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.647112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.10.11.194:37153)
0927 00:36:04.647228 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:04.647232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:04.647256 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:36:04.647452 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:04.647455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:04.647520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:04.647635 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:04.647921 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.648697 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.649965 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:04.650731 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:04.650849 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:04.651207 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:04.651762 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:04.651894 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:04.652102 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:36:04.652165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:36:04.659173 23439 raft_consensus.cc:1261] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 1 FOLLOWER]: Refusing update from remote peer e873cd962ef640799a3df8a504b019cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:04.659394 23189 raft_consensus.cc:1261] T bc67cc90f961459fa0ecf068374545c1 P d33991514c16433486a30cd5939686e4 [term 1 FOLLOWER]: Refusing update from remote peer e873cd962ef640799a3df8a504b019cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:04.660029 23545 consensus_queue.cc:1035] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5a6b5c6f9354387bcc73154cca5b81b" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:04.660498 23549 consensus_queue.cc:1035] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33991514c16433486a30cd5939686e4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:04.673331 23553 mvcc.cc:205] Tried to move back new op lower bound from 7333762517636341760 to 7333762517419708416. Current Snapshot: MvccSnapshot[applied={T|T < 7333762517636341760}]
I20260927 00:36:04.676970 23554 mvcc.cc:205] Tried to move back new op lower bound from 7333762517636341760 to 7333762517419708416. Current Snapshot: MvccSnapshot[applied={T|T < 7333762517636341760}]
I20260927 00:36:04.682008 23036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:04.682339 23036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:04.685706 23036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2886bf3ffc90477b89ff4edd37b2dd8b: Sending DeleteTablet for 3 replicas of tablet bc67cc90f961459fa0ecf068374545c1
I20260927 00:36:04.686637 23437 tablet_service.cc:1514] Processing DeleteTablet for tablet bc67cc90f961459fa0ecf068374545c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:04 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:59244
I20260927 00:36:04.687587 23303 tablet_service.cc:1514] Processing DeleteTablet for tablet bc67cc90f961459fa0ecf068374545c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:04 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:53500
I20260927 00:36:04.688669 23169 tablet_service.cc:1514] Processing DeleteTablet for tablet bc67cc90f961459fa0ecf068374545c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:04 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260927 00:36:04.689201 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23075
I20260927 00:36:04.694298 23562 tablet_replica.cc:330] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: stopping tablet replica
I20260927 00:36:04.694695 23562 raft_consensus.cc:2229] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:04.695118 23562 raft_consensus.cc:2258] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:04.696866 23562 ts_tablet_manager.cc:1905] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:04.697132 23563 tablet_replica.cc:330] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc: stopping tablet replica
I20260927 00:36:04.697449 23563 raft_consensus.cc:2229] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:36:04.697897 23563 raft_consensus.cc:2258] T bc67cc90f961459fa0ecf068374545c1 P e873cd962ef640799a3df8a504b019cc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:36:04.698663 23022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:44149 (error 108)
I20260927 00:36:04.700105 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.699775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44149 (local address 127.0.0.1:54604)
0927 00:36:04.699945 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.700015 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 00:36:04.700361 23022 catalog_manager.cc:4584] TS d33991514c16433486a30cd5939686e4 (127.10.11.193:44149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc67cc90f961459fa0ecf068374545c1: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44149: connect: Connection refused (error 111)
I20260927 00:36:04.700739 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23239
I20260927 00:36:04.701391 23562 ts_tablet_manager.cc:1918] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:04.701479 23562 log.cc:1196] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/bc67cc90f961459fa0ecf068374545c1
I20260927 00:36:04.701809 23562 ts_tablet_manager.cc:1939] T bc67cc90f961459fa0ecf068374545c1 P d5a6b5c6f9354387bcc73154cca5b81b: Deleting consensus metadata
I20260927 00:36:04.706187 23021 catalog_manager.cc:4841] TS d5a6b5c6f9354387bcc73154cca5b81b (127.10.11.195:36701): tablet bc67cc90f961459fa0ecf068374545c1 (table test-table [id=b012035df37347f3b3f296c82a7c5b1c]) successfully deleted
I20260927 00:36:04.708464 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:04.708183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:37153 (local address 127.0.0.1:53514)
0927 00:36:04.708309 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:04.708369 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.194:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:36:04.708608 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23369
I20260927 00:36:04.714505 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23004
I20260927 00:36:04.720212 10287 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:36:05.054436 10287 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:36:05.054538 10287 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:36:05.398625 10287 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:36:05.398 UTC [22071] LOG: received smart shutdown request
2026-09-27 00:36:05.402 UTC [22071] LOG: background worker "logical replication launcher" (PID 22080) exited with exit code 1
2026-09-27 00:36:05.402 UTC [22075] LOG: shutting down
2026-09-27 00:36:05.563 UTC [22071] LOG: database system is shut down
2026-09-27T00:36:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47772 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:36:05 dist-test-slave-6gxc krb5kdc[23576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:36:05 dist-test-slave-6gxc krb5kdc[23576](info): set up 2 sockets
Sep 27 00:36:05 dist-test-slave-6gxc krb5kdc[23576](info): commencing operation
krb5kdc: starting...
W20260927 00:36:06.710310 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:36:06 dist-test-slave-6gxc krb5kdc[23576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:36:06.827126 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39535
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:06.933723 23589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:06.933949 23589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:06.933987 23589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:06.937922 23589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:06.937971 23589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:06.937991 23589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:06.938009 23589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:06.938068 23589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:06.943106 23589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:06.944016 23589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:06.944947 23589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:06.952893 23597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:06.952945 23594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:06.952894 23595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:06.953217 23589 server_base.cc:1034] running on GCE node
I20260927 00:36:06.953730 23589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:06.954712 23589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:06.955883 23589 hybrid_clock.cc:648] HybridClock initialized: now 1790469366955871 us; error 25 us; skew 500 ppm
Sep 27 00:36:06 dist-test-slave-6gxc krb5kdc[23576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:06.958534 23589 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:06.959342 23589 webserver.cc:460] Webserver started at http://127.10.11.254:34481/ using document root <none> and password file <none>
I20260927 00:36:06.959803 23589 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:06.959860 23589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:06.960007 23589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:06.961815 23589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "a759552d832d49f191bfc6cec0749333"
format_stamp: "Formatted at 2026-09-27 00:36:06 on dist-test-slave-6gxc"
server_key: "b6742246736e9d1b0da947d9748f4648"
server_key_iv: "7770611cf8c7b791fd575b2ed6b4ea20"
server_key_version: "encryptionkey@0"
I20260927 00:36:06.962347 23589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "a759552d832d49f191bfc6cec0749333"
format_stamp: "Formatted at 2026-09-27 00:36:06 on dist-test-slave-6gxc"
server_key: "b6742246736e9d1b0da947d9748f4648"
server_key_iv: "7770611cf8c7b791fd575b2ed6b4ea20"
server_key_version: "encryptionkey@0"
I20260927 00:36:06.965552 23589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 00:36:06.967762 23604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:06.968942 23589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:36:06.969079 23589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "a759552d832d49f191bfc6cec0749333"
format_stamp: "Formatted at 2026-09-27 00:36:06 on dist-test-slave-6gxc"
server_key: "b6742246736e9d1b0da947d9748f4648"
server_key_iv: "7770611cf8c7b791fd575b2ed6b4ea20"
server_key_version: "encryptionkey@0"
I20260927 00:36:06.969180 23589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:06.984720 23589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:06.987702 23589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:06.987901 23589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:06.994763 23589 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:39535
I20260927 00:36:06.994781 23656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:39535 every 8 connection(s)
I20260927 00:36:06.995594 23589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:06.998020 23657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:07.001456 23657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:36:07.002383 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23589
I20260927 00:36:07.002506 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:36:07.002790 10287 external_mini_cluster.cc:1428] Setting key 9c5e086c5944b73127836df35ea56c62
WARNING: no policy specified for kudu/127.10.11.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.253@KRBTEST.COM" created.
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:07.019313 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.004643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.10.11.254:39535)
0927 00:36:07.004899 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.004909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.004940 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:36:07.005784 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.005788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.005948 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.006092 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.007162 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.008015 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.009561 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.009919 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.010031 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.014778 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.014791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.014800 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.014819 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.016401 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.016873 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.016878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.016880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.016978 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.017276 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.017282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.017284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.017614 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.017752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.018088 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.018384 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260927 00:36:07.019313 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.003803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39535 (local address 127.0.0.1:55368)
0927 00:36:07.004281 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.004354 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.005490 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.006169 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.006177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.006638 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.007005 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.007028 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.008148 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.008153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.009430 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.009442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.010078 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.010083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.010242 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.010782 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.010806 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.014622 (+ 3816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.016526 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.016535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.016543 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.016766 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.017077 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.017080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.017081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.017186 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.017728 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.017733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.017865 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.018020 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.018432 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":121,"thread_start_us":132,"threads_started":1}
I20260927 00:36:07.021454 23657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:36:07.022295 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.021901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32852)
0927 00:36:07.022074 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.022237 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 00:36:07.022507 23606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
I20260927 00:36:07.023420 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.023125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32868)
0927 00:36:07.023247 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.023345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 00:36:07.023586 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:40191: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
I20260927 00:36:07.028366 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:40191
--webserver_interface=127.10.11.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:07.072803 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } attempt: 1
I20260927 00:36:07.073644 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.073319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32880)
0927 00:36:07.073485 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.073556 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:36:07.074503 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.074255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32888)
0927 00:36:07.074386 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.074441 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:36:07.074674 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:40191: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
W20260927 00:36:07.132936 23664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:07.133173 23664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:07.133253 23664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:07.137243 23664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:07.137292 23664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:07.137311 23664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:07.137327 23664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:07.137382 23664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:07.142373 23664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.11.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:07.142833 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } attempt: 2
I20260927 00:36:07.143426 23664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:07.143646 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.143325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32894)
0927 00:36:07.143494 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.143561 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:36:07.144680 23664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:36:07.144824 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.144556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32904)
0927 00:36:07.144686 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.144760 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 00:36:07.144928 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:40191: Network error: Client connection negotiation failed: client connection to 127.10.11.253:40191: connect: Connection refused (error 111)
W20260927 00:36:07.152947 23672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:07.153031 23670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:07.153087 23664 server_base.cc:1034] running on GCE node
W20260927 00:36:07.152947 23669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:07.153793 23664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:07.154842 23664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:07.155994 23664 hybrid_clock.cc:648] HybridClock initialized: now 1790469367155977 us; error 27 us; skew 500 ppm
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:07.158609 23664 init.cc:377] Logged in from keytab as kudu/127.10.11.253@KRBTEST.COM (short username kudu)
I20260927 00:36:07.159456 23664 webserver.cc:460] Webserver started at http://127.10.11.253:41181/ using document root <none> and password file <none>
I20260927 00:36:07.159986 23664 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:07.160044 23664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:07.160195 23664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:07.162009 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data/instance:
uuid: "5ccf9338ed6e4090a9be4869aaeff11f"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "4614660335fb245885e8a994139aad17"
server_key_iv: "dfc1c35b0f04627e4ac259d11409debf"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.162560 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal/instance:
uuid: "5ccf9338ed6e4090a9be4869aaeff11f"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "4614660335fb245885e8a994139aad17"
server_key_iv: "dfc1c35b0f04627e4ac259d11409debf"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.165918 23664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:36:07.168195 23679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:07.169342 23664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:07.169478 23664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "5ccf9338ed6e4090a9be4869aaeff11f"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "4614660335fb245885e8a994139aad17"
server_key_iv: "dfc1c35b0f04627e4ac259d11409debf"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.169581 23664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:07.188273 23664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:07.191056 23664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:07.191247 23664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:07.198326 23664 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.253:40191
I20260927 00:36:07.198345 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.253:40191 every 8 connection(s)
I20260927 00:36:07.199199 23664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
I20260927 00:36:07.201684 23732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:07.203464 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23664
I20260927 00:36:07.203603 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-1/wal/instance
I20260927 00:36:07.203924 10287 external_mini_cluster.cc:1428] Setting key 6c3e4c291fd10e72afc283be39b0873d
I20260927 00:36:07.205406 23732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.11.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.252@KRBTEST.COM" created.
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:07.219172 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.207189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.10.11.254:39535)
0927 00:36:07.207318 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.207322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.208244 (+ 922us) server_negotiation.cc:407] Connection header received
0927 00:36:07.209111 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.209114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.209175 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.209271 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.210366 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.210882 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.212031 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.212283 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.212372 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.214886 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.214889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.214892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.214919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.216717 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.217280 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.217287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.217291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.217400 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.217791 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.217798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.217803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.218143 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.218265 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.218792 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.218935 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:36:07.219656 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.207619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39535 (local address 127.0.0.1:55382)
0927 00:36:07.208097 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.208135 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.208907 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.209426 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.209433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.209742 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.210231 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.210245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.211004 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.211008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.211890 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.211899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.212413 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.212416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.212545 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.213090 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.213106 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.214695 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.216850 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.216855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.216864 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.217108 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.217531 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.217534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.217535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.217663 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.218284 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.218289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.218497 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.218670 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.218895 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":150,"threads_started":1}
I20260927 00:36:07.220947 23732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
I20260927 00:36:07.229228 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:44721
--webserver_interface=127.10.11.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:07.229809 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.221378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32920)
0927 00:36:07.221479 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.221493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.221562 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.222186 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.222189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.222309 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.222411 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.222421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.223489 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.223492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.224055 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.224064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.224701 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.224704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.224787 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.225192 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.225204 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.226437 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.228226 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.228229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.228230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.228509 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.228803 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.228809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.228812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.228887 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.229315 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.229318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.229402 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.229501 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.229615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 00:36:07.230022 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.221527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.10.11.253:40191)
0927 00:36:07.221819 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.221832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.221864 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:36:07.221898 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.221902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.222035 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.222205 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.222551 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.223381 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.224193 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.224582 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.224670 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.226556 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.226564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.226573 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.226592 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.228110 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.228615 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.228620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.228624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.228701 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.228980 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.228983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.228985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.229208 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.229348 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.229591 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.229703 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260927 00:36:07.231848 23732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:36:07.232456 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.232245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52430)
0927 00:36:07.232347 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.232407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:36:07.232640 23681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.233345 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.233149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52440)
0927 00:36:07.233247 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.233298 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:36:07.233459 23732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.236091 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } attempt: 3
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:36:07.244822 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.236444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:32930)
0927 00:36:07.236598 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.236615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.236710 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.236948 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.236953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.237146 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.237288 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.237299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.237976 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.237983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.239478 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.239507 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.239935 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.239938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.240057 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.240455 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.240469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.241748 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.243598 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.243600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.243602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.243794 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.244048 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.244050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.244051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.244120 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.244465 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.244467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.244540 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.244615 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.244716 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:36:07.244930 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.236549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.10.11.253:40191)
0927 00:36:07.236644 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.236647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.236658 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:36:07.236803 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.236805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.236838 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.236899 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.237411 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.237874 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.239607 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.239834 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.239910 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.241891 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.241895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.241897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.241921 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.243492 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.243882 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.243885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.243887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.243963 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.244198 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.244201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.244203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.244383 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.244470 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.244690 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.244789 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:36:07.245966 23657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:36:07.246610 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.246413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52442)
0927 00:36:07.246515 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.246559 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:36:07.247251 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.247021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52458)
0927 00:36:07.247141 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.247189 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 00:36:07.247395 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.278563 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 1
I20260927 00:36:07.279181 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.278952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52472)
0927 00:36:07.279070 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.279117 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:36:07.279841 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.279633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52480)
0927 00:36:07.279750 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.279794 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:36:07.279978 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.282662 23732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 1
I20260927 00:36:07.283289 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.283021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52492)
0927 00:36:07.283128 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.283223 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:36:07.283917 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.283715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:52496)
0927 00:36:07.283817 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.283866 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:36:07.284021 23732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
W20260927 00:36:07.337133 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:07.337383 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:07.337462 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:07.341315 23738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:07.341391 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:07.341427 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:07.341485 23738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:07.341524 23738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:07.346304 23738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39535,127.10.11.253:40191,127.10.11.252:44721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:44721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.11.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:07.347157 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:07.348073 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:36:07.352170 23732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 2
I20260927 00:36:07.352919 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.352656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40252)
0927 00:36:07.352795 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.352850 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:36:07.353578 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.353366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40260)
0927 00:36:07.353470 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.353517 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:36:07.353711 23732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.355129 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 2
I20260927 00:36:07.355887 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.355533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40266)
0927 00:36:07.355732 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.355801 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 00:36:07.356345 23745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:07.356480 23747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:07.356578 23738 server_base.cc:1034] running on GCE node
W20260927 00:36:07.356678 23744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:07.356897 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.356514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40268)
0927 00:36:07.356772 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.356827 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260927 00:36:07.357025 23657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44721: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44721: connect: Connection refused (error 111)
I20260927 00:36:07.357290 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:07.358399 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:07.359582 23738 hybrid_clock.cc:648] HybridClock initialized: now 1790469367359569 us; error 26 us; skew 500 ppm
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:07.362290 23738 init.cc:377] Logged in from keytab as kudu/127.10.11.252@KRBTEST.COM (short username kudu)
I20260927 00:36:07.363137 23738 webserver.cc:460] Webserver started at http://127.10.11.252:35437/ using document root <none> and password file <none>
I20260927 00:36:07.363659 23738 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:07.363762 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:07.363958 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:07.365856 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data/instance:
uuid: "6091ff8b9cd1454485994068044300fc"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "cc960f16280e7755b5835486f230072b"
server_key_iv: "821babb103cf4f4eb0bc0c34649a493a"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.366523 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal/instance:
uuid: "6091ff8b9cd1454485994068044300fc"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "cc960f16280e7755b5835486f230072b"
server_key_iv: "821babb103cf4f4eb0bc0c34649a493a"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.369904 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:36:07.372226 23754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:07.373332 23738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:07.373467 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "6091ff8b9cd1454485994068044300fc"
format_stamp: "Formatted at 2026-09-27 00:36:07 on dist-test-slave-6gxc"
server_key: "cc960f16280e7755b5835486f230072b"
server_key_iv: "821babb103cf4f4eb0bc0c34649a493a"
server_key_version: "encryptionkey@0"
I20260927 00:36:07.373621 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:07.381966 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:07.384629 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:07.384822 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:07.391757 23738 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.252:44721
I20260927 00:36:07.391769 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.252:44721 every 8 connection(s)
I20260927 00:36:07.392617 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
I20260927 00:36:07.394824 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23738
I20260927 00:36:07.394959 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469213433099-10287-0/minicluster-data/master-2/wal/instance
I20260927 00:36:07.395299 10287 external_mini_cluster.cc:1428] Setting key e6bc253c02245d7f9fa97eacd81a2d01
I20260927 00:36:07.395357 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:07.399632 23807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:07.409119 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.396585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.10.11.252:44721)
0927 00:36:07.396983 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.396997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.397028 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:36:07.397902 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.397919 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.398244 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.398545 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.399349 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.400334 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.401396 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.401958 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.402153 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.404737 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.404743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.404752 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.404776 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.406516 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.406944 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.406956 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.406960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.407053 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.407330 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.407335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.407336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.407795 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.408029 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.408261 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.408497 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260927 00:36:07.410421 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.401215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.10.11.254:39535)
0927 00:36:07.401344 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.401349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.401623 (+ 274us) server_negotiation.cc:407] Connection header received
0927 00:36:07.401791 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.401795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.401856 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.401952 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.402800 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.403296 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.404584 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.404814 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.404934 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.407051 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.407056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.407058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.407209 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.408764 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.409204 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.409208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.409210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.409294 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.409558 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.409561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.409562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.409739 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.409843 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.410173 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.410274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:36:07.410437 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.401133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39535 (local address 127.0.0.1:44926)
0927 00:36:07.401488 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.401524 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.401686 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.401998 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.402004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.402466 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.402673 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.402686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.403448 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.403455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.404454 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.404464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.404934 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.404938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.405065 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.405530 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.405543 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.406905 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.408866 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.408876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.408881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.409098 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.409371 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.409376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.409378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.409460 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.409842 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.409849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.409955 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.410085 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.410275 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
I20260927 00:36:07.411408 23807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:07.419561 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.411838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:40191 (local address 127.0.0.1:52608)
0927 00:36:07.411973 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.411988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.412051 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.412262 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.412268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.412443 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.412564 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.412570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.413237 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.413250 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.413728 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.413735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.414196 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.414200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.414300 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.414727 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.414744 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.416144 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.418150 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.418153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.418155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.418360 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.418616 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.418619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.418621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.418718 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.419107 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.419109 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.419199 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.419277 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.419402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 00:36:07.419838 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.411345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.10.11.254:39535)
0927 00:36:07.411756 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.411762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.411934 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:36:07.412070 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.412076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.412122 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.412177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.412958 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.413416 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.414163 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.414389 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.414495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.416810 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.416816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.416820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.416847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.418300 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.418712 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.418715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.418716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.418807 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.419052 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.419056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.419059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.419240 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.419461 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.419587 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.419700 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260927 00:36:07.419953 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.411947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.10.11.253:40191)
0927 00:36:07.412049 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.412053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.412064 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:36:07.412125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.412127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.412163 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.412226 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.412675 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.413132 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.413851 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.414065 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.414198 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.416295 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.416298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.416300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.416339 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.417998 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.418458 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.418460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.418462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.418529 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.418808 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.418812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.418814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.418994 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.419109 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.419357 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.419465 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 00:36:07.420608 23807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:36:07.428437 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.421143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40278)
0927 00:36:07.421295 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.421308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.421372 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.421565 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.421567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.421721 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.421772 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.421777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.422522 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.422525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.422999 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.423006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.423429 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.423433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.423503 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.423864 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.423874 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.425127 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.426990 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.426996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.427000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.427244 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.427512 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.427517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.427522 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.427613 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.427969 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.427972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.428051 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.428151 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.428281 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 00:36:07.428463 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.421217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.10.11.252:44721)
0927 00:36:07.421316 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.421321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.421339 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:36:07.421439 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.421442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.421473 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.421556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.421874 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.422421 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.423106 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.423333 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.423431 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.425270 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.425273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.425276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.425299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.426866 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.427354 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.427356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.427357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.427425 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.427715 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.427717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.427718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.427868 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.427978 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.428206 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.428336 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":45}
I20260927 00:36:07.429316 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.420931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.10.11.253:40191)
0927 00:36:07.421049 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.421053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.421066 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:36:07.421169 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.421175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.421215 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.421290 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.421876 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.422423 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.423127 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.423349 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.423448 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.426253 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.426257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.426263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.426286 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.427768 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.428172 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.428175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.428177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.428254 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.428521 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.428524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.428525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.428735 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.428974 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.429058 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.429183 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:36:07.431679 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: Bootstrap starting.
I20260927 00:36:07.432473 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:36:07.433938 23807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:07.434775 23807 log.cc:824] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:07.436506 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: No bootstrap required, opened a new log
I20260927 00:36:07.440418 23807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.440696 23807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:07.440749 23807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6091ff8b9cd1454485994068044300fc, State: Initialized, Role: FOLLOWER
I20260927 00:36:07.441212 23807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.442553 23807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:07.443976 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.434649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.10.11.253:40191)
0927 00:36:07.435250 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.435256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.435273 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:36:07.435382 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.435385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.435430 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.435476 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.436428 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.437100 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.438138 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.438450 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.438541 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.440278 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.440281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.440284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.440329 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.441986 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.442524 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.442527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.442529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.442604 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.442910 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.442914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.442915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.443096 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.443202 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.443659 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.443798 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260927 00:36:07.444581 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.435172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.10.11.254:39535)
0927 00:36:07.435273 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.435277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.435290 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:36:07.435326 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.435331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.435375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.435440 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.436444 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.436882 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.437975 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.438352 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.438451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.440062 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.440066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.440068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.440091 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.442085 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.442688 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.442691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.442693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.442773 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.443076 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.443080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.443081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.443283 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.443420 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.444319 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.444437 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:36:07.445070 23732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 3
I20260927 00:36:07.448164 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.434968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.10.11.252:44721)
0927 00:36:07.435065 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.435072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.438167 (+ 3095us) server_negotiation.cc:407] Connection header received
0927 00:36:07.438316 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.438319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.438356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.438402 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.439822 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.441476 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.442760 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.443071 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.443147 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.444792 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.444796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.444798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.444820 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.446440 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.446899 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.446902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.446904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.446987 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.447298 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.447301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.447303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.447484 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.447568 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.447920 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.448020 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 00:36:07.448647 23831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:36:07.448714 23831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:36:07.446957 23738 master_runner.cc:186] Error getting master registration for 127.10.11.254:39535: OK, instance_id { permanent_uuid: "a759552d832d49f191bfc6cec0749333" instance_seqno: 1790469366992838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:36:07.450251 23738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469367, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:36:07.454470 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.445430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40294)
0927 00:36:07.445531 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.445544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.445617 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.446428 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.446431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.446593 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.446692 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.446699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.447804 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.447807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.448612 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.448622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.449439 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.449444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.449522 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.449987 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.450000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.451192 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.452995 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.452998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.452999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.453189 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.453471 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.453473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.453474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.453588 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.454002 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.454005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.454088 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.454199 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.454317 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 00:36:07.455089 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.445552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.10.11.252:44721)
0927 00:36:07.446206 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.446215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.446226 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:36:07.446257 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.446259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.446293 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.446803 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.446924 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.447679 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.448984 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.449306 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.449733 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.451328 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.451332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.451335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.451357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.452898 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.453282 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.453286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.453287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.453382 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.453690 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.453694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.453696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.453883 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.454729 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.454813 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.454915 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":92,"threads_started":1}
I20260927 00:36:07.456184 23657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } attempt: 3
I20260927 00:36:07.457927 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: Bootstrap starting.
I20260927 00:36:07.458287 23820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.458410 23820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.460150 23732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:07.461009 23732 log.cc:824] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:36:07 dist-test-slave-6gxc krb5kdc[23576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:36:07.463007 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: No bootstrap required, opened a new log
I20260927 00:36:07.465692 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.456491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44721 (local address 127.0.0.1:40300)
0927 00:36:07.456630 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:07.456643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:07.456714 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:07.456970 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:07.456973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:07.457141 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:07.457247 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.457253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.458731 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.458735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.459534 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:07.459544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.460181 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.460184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:07.460266 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.460717 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:07.460731 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:07.462236 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:07.464183 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.464187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.464190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.464421 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.464745 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:07.464748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:07.464750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:07.464841 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:07.465221 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:07.465224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:07.465312 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:36:07.465421 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:36:07.465536 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 00:36:07.465732 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:07.456561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.10.11.252:44721)
0927 00:36:07.456658 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:07.456662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:07.456677 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:36:07.456803 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:07.456806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:07.456842 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:07.456922 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:07.457371 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.458590 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.459689 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:07.460048 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:07.460146 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:07.462373 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:07.462377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:07.462379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:07.462403 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:07.464060 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.464533 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.464536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.464537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.464633 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:07.464937 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:07.464940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:07.464942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:07.465121 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:07.465212 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:07.465482 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 00:36:07.465587 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:36:07.466890 23732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.467084 23732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:07.467144 23732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf9338ed6e4090a9be4869aaeff11f, State: Initialized, Role: FOLLOWER
I20260927 00:36:07.467559 23732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.468025 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.468199 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.468315 23657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: Bootstrap starting.
I20260927 00:36:07.468763 23732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:07.470719 23657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:07.471519 23657 log.cc:824] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:07.473320 23657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: No bootstrap required, opened a new log
W20260927 00:36:07.474057 23846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:36:07.474133 23846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:36:07.471989 23664 master_runner.cc:186] Error getting master registration for 127.10.11.254:39535: OK, instance_id { permanent_uuid: "a759552d832d49f191bfc6cec0749333" instance_seqno: 1790469366992838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:36:07.475092 23664 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:36:07.477030 23657 raft_consensus.cc:348] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.477224 23657 raft_consensus.cc:374] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:07.477273 23657 raft_consensus.cc:729] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a759552d832d49f191bfc6cec0749333, State: Initialized, Role: FOLLOWER
I20260927 00:36:07.477674 23657 consensus_queue.cc:260] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.478184 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.478370 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.478848 23657 sys_catalog.cc:564] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:07.486462 23589 master_runner.cc:263] No leader master found from master a759552d832d49f191bfc6cec0749333
I20260927 00:36:07.486531 23589 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:36:07.489208 23610 tablet.cc:2369] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:07.491802 23860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:36:07.491940 23860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 00:36:07.636015 23760 tablet.cc:2369] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:07.692672 23685 tablet.cc:2369] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:07.807020 23820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:07.807204 23820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.808192 23820 leader_election.cc:288] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a759552d832d49f191bfc6cec0749333 (127.10.11.254:39535), 5ccf9338ed6e4090a9be4869aaeff11f (127.10.11.253:40191)
I20260927 00:36:07.808715 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6091ff8b9cd1454485994068044300fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a759552d832d49f191bfc6cec0749333" is_pre_election: true
I20260927 00:36:07.808774 23706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6091ff8b9cd1454485994068044300fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" is_pre_election: true
I20260927 00:36:07.809017 23631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6091ff8b9cd1454485994068044300fc in term 0.
I20260927 00:36:07.809111 23706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6091ff8b9cd1454485994068044300fc in term 0.
I20260927 00:36:07.809456 23756 leader_election.cc:302] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6091ff8b9cd1454485994068044300fc, a759552d832d49f191bfc6cec0749333; no voters:
I20260927 00:36:07.809731 23820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:07.809839 23820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:07.809899 23820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:07.810941 23820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.811270 23820 leader_election.cc:288] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [CANDIDATE]: Term 1 election: Requested vote from peers a759552d832d49f191bfc6cec0749333 (127.10.11.254:39535), 5ccf9338ed6e4090a9be4869aaeff11f (127.10.11.253:40191)
I20260927 00:36:07.811700 23706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6091ff8b9cd1454485994068044300fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf9338ed6e4090a9be4869aaeff11f"
I20260927 00:36:07.811700 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6091ff8b9cd1454485994068044300fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a759552d832d49f191bfc6cec0749333"
I20260927 00:36:07.811828 23706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:07.811828 23631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:07.813093 23631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6091ff8b9cd1454485994068044300fc in term 1.
I20260927 00:36:07.813092 23706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6091ff8b9cd1454485994068044300fc in term 1.
I20260927 00:36:07.813573 23756 leader_election.cc:302] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ccf9338ed6e4090a9be4869aaeff11f, 6091ff8b9cd1454485994068044300fc; no voters:
I20260927 00:36:07.813724 23820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:07.813908 23820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [term 1 LEADER]: Becoming Leader. State: Replica: 6091ff8b9cd1454485994068044300fc, State: Running, Role: LEADER
I20260927 00:36:07.814292 23820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } }
I20260927 00:36:07.816124 23862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6091ff8b9cd1454485994068044300fc. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.816236 23862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:07.816572 23864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:07.817366 23864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:07.821540 23631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [term 1 FOLLOWER]: Refusing update from remote peer 6091ff8b9cd1454485994068044300fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:07.821574 23706 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [term 1 FOLLOWER]: Refusing update from remote peer 6091ff8b9cd1454485994068044300fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:07.822187 23862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:07.822326 23820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:07.825970 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6091ff8b9cd1454485994068044300fc. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.825994 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6091ff8b9cd1454485994068044300fc. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.826097 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.826121 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.828383 23820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.828416 23862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.828502 23820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:07.828540 23865 mvcc.cc:205] Tried to move back new op lower bound from 7333762530591092736 to 7333762530569777152. Current Snapshot: MvccSnapshot[applied={T|T < 7333762530591092736}]
I20260927 00:36:07.828512 23862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:07.828668 23867 mvcc.cc:205] Tried to move back new op lower bound from 7333762530591092736 to 7333762530569777152. Current Snapshot: MvccSnapshot[applied={T|T < 7333762530591092736}]
I20260927 00:36:07.828825 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.828930 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.829593 23864 catalog_manager.cc:1347] Generated new cluster ID: 50aff4934a2f43f0b96c809cf7c707d7
I20260927 00:36:07.829646 23864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:07.829627 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6091ff8b9cd1454485994068044300fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759552d832d49f191bfc6cec0749333" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39535 } } peers { permanent_uuid: "5ccf9338ed6e4090a9be4869aaeff11f" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 40191 } } peers { permanent_uuid: "6091ff8b9cd1454485994068044300fc" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44721 } } }
I20260927 00:36:07.829728 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:36:07.829783 23866 mvcc.cc:205] Tried to move back new op lower bound from 7333762530591092736 to 7333762530569777152. Current Snapshot: MvccSnapshot[applied={T|T < 7333762530591092736}]
I20260927 00:36:07.850704 23864 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:36:07.851912 23864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:07.858605 23864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6091ff8b9cd1454485994068044300fc: Generated new TSK 0
I20260927 00:36:07.859395 23864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:36:08.475464 23846 catalog_manager.cc:1259] Loaded cluster ID: 50aff4934a2f43f0b96c809cf7c707d7
I20260927 00:36:08.475553 23846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: loading cluster ID for follower catalog manager: success
I20260927 00:36:08.477314 23846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: acquiring CA information for follower catalog manager: success
I20260927 00:36:08.478631 23846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ccf9338ed6e4090a9be4869aaeff11f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:36:08.492964 23860 catalog_manager.cc:1259] Loaded cluster ID: 50aff4934a2f43f0b96c809cf7c707d7
I20260927 00:36:08.493021 23860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: loading cluster ID for follower catalog manager: success
I20260927 00:36:08.494485 23860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: acquiring CA information for follower catalog manager: success
I20260927 00:36:08.495581 23860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a759552d832d49f191bfc6cec0749333: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:36:08.920410 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23589
I20260927 00:36:08.926555 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23664
I20260927 00:36:08.932199 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23738
2026-09-27T00:36:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3249 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:36:08 dist-test-slave-6gxc krb5kdc[23881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:36:08 dist-test-slave-6gxc krb5kdc[23881](info): set up 2 sockets
Sep 27 00:36:08 dist-test-slave-6gxc krb5kdc[23881](info): commencing operation
krb5kdc: starting...
W20260927 00:36:09.966199 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:36:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:36:10.080621 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41789
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:10.186484 23894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:10.186715 23894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:10.186753 23894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:10.190665 23894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:10.190709 23894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:10.190728 23894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:10.190744 23894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:10.190797 23894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:10.195856 23894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:41789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:10.196779 23894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:10.197729 23894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:10.205595 23900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:10.205590 23899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:10.205735 23894 server_base.cc:1034] running on GCE node
W20260927 00:36:10.205823 23902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:10.206519 23894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:10.207485 23894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:10.208719 23894 hybrid_clock.cc:648] HybridClock initialized: now 1790469370208687 us; error 41 us; skew 500 ppm
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469370, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:10.211440 23894 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:10.212170 23894 webserver.cc:460] Webserver started at http://127.10.11.254:40123/ using document root <none> and password file <none>
I20260927 00:36:10.212663 23894 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:10.212719 23894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:10.212868 23894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:10.214745 23894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "9fc479a7cae54319bbe4af71eebd63ec"
format_stamp: "Formatted at 2026-09-27 00:36:10 on dist-test-slave-6gxc"
server_key: "cd89c69901367569f31b1b9a1a0d2a1a"
server_key_iv: "0e3189a2aaa2d5401e618220065ff2f1"
server_key_version: "encryptionkey@0"
I20260927 00:36:10.215251 23894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "9fc479a7cae54319bbe4af71eebd63ec"
format_stamp: "Formatted at 2026-09-27 00:36:10 on dist-test-slave-6gxc"
server_key: "cd89c69901367569f31b1b9a1a0d2a1a"
server_key_iv: "0e3189a2aaa2d5401e618220065ff2f1"
server_key_version: "encryptionkey@0"
I20260927 00:36:10.218660 23894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:36:10.220798 23909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:10.221910 23894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:10.222005 23894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "9fc479a7cae54319bbe4af71eebd63ec"
format_stamp: "Formatted at 2026-09-27 00:36:10 on dist-test-slave-6gxc"
server_key: "cd89c69901367569f31b1b9a1a0d2a1a"
server_key_iv: "0e3189a2aaa2d5401e618220065ff2f1"
server_key_version: "encryptionkey@0"
I20260927 00:36:10.222085 23894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:10.246017 23894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:10.248984 23894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:10.249136 23894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:10.255508 23894 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:41789
I20260927 00:36:10.255527 23961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:41789 every 8 connection(s)
I20260927 00:36:10.256414 23894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:10.258944 23962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:10.264022 23962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: Bootstrap starting.
I20260927 00:36:10.266049 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 23894
I20260927 00:36:10.266209 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:36:10.266227 23962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:10.266515 10287 external_mini_cluster.cc:1428] Setting key e7a3ecb32b1c5f43d93131b030270030
I20260927 00:36:10.266968 23962 log.cc:824] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:10.269048 23962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: No bootstrap required, opened a new log
I20260927 00:36:10.273147 23962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } }
I20260927 00:36:10.273396 23962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:10.273492 23962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc479a7cae54319bbe4af71eebd63ec, State: Initialized, Role: FOLLOWER
I20260927 00:36:10.273928 23962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } }
I20260927 00:36:10.274041 23962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:10.274104 23962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:10.274217 23962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:10.274986 23962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } }
I20260927 00:36:10.275296 23962 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc479a7cae54319bbe4af71eebd63ec; no voters:
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:10.275599 23962 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:36:10.275677 23967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:10.275887 23967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [term 1 LEADER]: Becoming Leader. State: Replica: 9fc479a7cae54319bbe4af71eebd63ec, State: Running, Role: LEADER
I20260927 00:36:10.276266 23967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } }
I20260927 00:36:10.276777 23962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:10.278045 23968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc479a7cae54319bbe4af71eebd63ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } } }
I20260927 00:36:10.278190 23968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:10.278537 23969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc479a7cae54319bbe4af71eebd63ec. Latest consensus state: current_term: 1 leader_uuid: "9fc479a7cae54319bbe4af71eebd63ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc479a7cae54319bbe4af71eebd63ec" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41789 } } }
I20260927 00:36:10.278693 23969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec [sys.catalog]: This master's current role is: LEADER
W20260927 00:36:10.281253 23982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:36:10.281279 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:10.267408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.10.11.254:41789)
0927 00:36:10.267853 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:10.267889 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:10.267926 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:36:10.268679 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:10.268700 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:10.269057 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:10.269449 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:10.270142 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:10.271642 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:10.272502 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:10.273172 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:10.273334 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:10.276091 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:10.276097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:10.276111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:10.276139 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:10.278278 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:10.278808 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:10.278816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:10.278824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:10.278953 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:10.279265 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:10.279270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:10.279271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:10.279687 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:10.279836 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:10.280162 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 00:36:10.280426 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":108,"threads_started":1}
W20260927 00:36:10.281353 23982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:36:10.281576 23977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:10.282403 23977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:10.284143 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:36:10.288503 23977 catalog_manager.cc:1347] Generated new cluster ID: 1ce681aa90594686ab99bb8c72e48b17
I20260927 00:36:10.288560 23977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:10.305503 23977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:36:10.306340 23977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:10.312958 23977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc479a7cae54319bbe4af71eebd63ec: Generated new TSK 0
I20260927 00:36:10.313522 23977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:36:10.339421 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 23894
2026-09-27T00:36:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1404 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23989](info): set up 2 sockets
Sep 27 00:36:10 dist-test-slave-6gxc krb5kdc[23989](info): commencing operation
krb5kdc: starting...
W20260927 00:36:11.374346 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:36:11 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:36:11.489342 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:11.594044 24002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:11.594312 24002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:11.594381 24002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:11.598277 24002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:11.598346 24002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:11.598383 24002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:11.598419 24002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:11.598454 24002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:11.604624 24002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:11.605525 24002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:11.606488 24002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:11.614449 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:11.614441 24007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:11.614441 24008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:11.614799 24002 server_base.cc:1034] running on GCE node
I20260927 00:36:11.615345 24002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:11.616285 24002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:11.617455 24002 hybrid_clock.cc:648] HybridClock initialized: now 1790469371617407 us; error 58 us; skew 500 ppm
Sep 27 00:36:11 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469371, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:11.625272 24002 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:11.626281 24002 webserver.cc:460] Webserver started at http://127.10.11.254:45373/ using document root <none> and password file <none>
I20260927 00:36:11.626824 24002 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:11.626914 24002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:11.627136 24002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:11.629041 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "9b99071e7c9241738721192ac327146d"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "baf0c94ebae918dadf4ab010d49ec3b5"
server_key_iv: "4f4b61c9fb04ff5240926c884e93ddcd"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.629586 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "9b99071e7c9241738721192ac327146d"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "baf0c94ebae918dadf4ab010d49ec3b5"
server_key_iv: "4f4b61c9fb04ff5240926c884e93ddcd"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.633126 24002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:36:11.635578 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:11.636806 24002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:11.636961 24002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "9b99071e7c9241738721192ac327146d"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "baf0c94ebae918dadf4ab010d49ec3b5"
server_key_iv: "4f4b61c9fb04ff5240926c884e93ddcd"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.637072 24002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:11.653550 24002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:11.656394 24002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:11.656592 24002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:11.663550 24002 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:38669
I20260927 00:36:11.663825 24069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:38669 every 8 connection(s)
I20260927 00:36:11.664542 24002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:11.667140 24070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:11.672294 24070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap starting.
I20260927 00:36:11.674448 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24002
I20260927 00:36:11.674579 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:36:11.674783 24070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:11.674911 10287 external_mini_cluster.cc:1428] Setting key 90dae36490c332f0f5609a3afeb4e99f
I20260927 00:36:11.675572 24070 log.cc:824] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:11.677479 24070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: No bootstrap required, opened a new log
I20260927 00:36:11.681486 24070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:11.681730 24070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:11.681774 24070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b99071e7c9241738721192ac327146d, State: Initialized, Role: FOLLOWER
I20260927 00:36:11.682319 24070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:11.682495 24070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:11.682575 24070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:11.682698 24070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:36:11 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:11.683696 24070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:11.684027 24070 leader_election.cc:302] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b99071e7c9241738721192ac327146d; no voters:
I20260927 00:36:11.684337 24070 leader_election.cc:288] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:36:11.684445 24075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:11.684933 24075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 LEADER]: Becoming Leader. State: Replica: 9b99071e7c9241738721192ac327146d, State: Running, Role: LEADER
I20260927 00:36:11.685300 24075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:11.685693 24070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:11.686908 24077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b99071e7c9241738721192ac327146d. Latest consensus state: current_term: 1 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:11.687036 24077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:11.687369 24076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:11.687467 24076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:11.689776 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:11.676012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.10.11.254:38669)
0927 00:36:11.676376 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:11.676391 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:11.676426 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:36:11.677019 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:11.677035 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:11.677327 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:11.677644 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:11.678487 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.679428 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.680670 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.681107 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.681219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:11.684180 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:11.684186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:11.684203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:11.684235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:11.686313 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:11.686910 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:11.686918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:11.686926 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:11.687076 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:11.687415 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:11.687419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:11.687422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:11.687970 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:11.688185 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:11.688540 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:36:11.688845 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
W20260927 00:36:11.690609 24090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:36:11.690696 24090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:36:11.690802 24084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:11.691607 24084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:11.696781 24084 catalog_manager.cc:1347] Generated new cluster ID: d8e8af573eae4e3ab91c58239dd3496b
I20260927 00:36:11.696839 24084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:11.714553 24084 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:36:11.715718 24084 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:11.721558 24084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Generated new TSK 0
I20260927 00:36:11.722270 24084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:36:11.756392 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:11.876631 24096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:11.876863 24096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:11.876901 24096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:11.880698 24096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:11.880745 24096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:11.880798 24096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:11.885604 24096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:11.886597 24096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:11.887621 24096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:11.896649 24101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:11.896688 24102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:11.896816 24096 server_base.cc:1034] running on GCE node
W20260927 00:36:11.896687 24104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:11.897279 24096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:11.897902 24096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:11.899085 24096 hybrid_clock.cc:648] HybridClock initialized: now 1790469371899086 us; error 48 us; skew 500 ppm
Sep 27 00:36:11 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469371, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:11.901808 24096 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:11.902639 24096 webserver.cc:460] Webserver started at http://127.10.11.193:33951/ using document root <none> and password file <none>
I20260927 00:36:11.903275 24096 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:11.903360 24096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:11.903560 24096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:11.905427 24096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "601c11ab81954f2095240a3615e054b6"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "2d710ebf1c27025114635b390110b2eb"
server_key_iv: "52660eba66cfacc694e3014d1b421900"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.906003 24096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "601c11ab81954f2095240a3615e054b6"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "2d710ebf1c27025114635b390110b2eb"
server_key_iv: "52660eba66cfacc694e3014d1b421900"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.909513 24096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:11.911932 24111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:11.913087 24096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:36:11.913239 24096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "601c11ab81954f2095240a3615e054b6"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "2d710ebf1c27025114635b390110b2eb"
server_key_iv: "52660eba66cfacc694e3014d1b421900"
server_key_version: "encryptionkey@0"
I20260927 00:36:11.913349 24096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:11.923916 24096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:11.926532 24096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:11.926723 24096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:11.927250 24096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:11.928020 24096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:11.928097 24096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:11.928184 24096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:11.928238 24096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:11.936738 24096 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:42659
I20260927 00:36:11.936756 24224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:42659 every 8 connection(s)
I20260927 00:36:11.937532 24096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:11.942406 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24096
I20260927 00:36:11.942533 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:36:11.942834 10287 external_mini_cluster.cc:1428] Setting key 075b2495360d287b3e4971132b3a98c1
Sep 27 00:36:11 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469371, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:36:11.951432 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:11.939351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:54811 (local address 127.10.11.254:38669)
0927 00:36:11.939513 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:11.939518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:11.940332 (+ 814us) server_negotiation.cc:407] Connection header received
0927 00:36:11.941718 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:11.941721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:11.941767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:11.941839 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:11.943258 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.943839 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.944959 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.945217 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.945330 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:11.947958 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:11.947968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:11.947971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:11.947993 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:11.949438 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:11.949931 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:11.949936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:11.949938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:11.950018 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:11.950375 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:11.950380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:11.950382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:11.950557 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:11.950662 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:11.951146 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 00:36:11.951265 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:36:11.952200 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:11.939734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:54811)
0927 00:36:11.940176 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:11.940221 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:11.941521 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:11.941956 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:11.941962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:11.942444 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:11.943093 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:11.943113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.943969 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.943973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:11.944830 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:11.944839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:11.945367 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:11.945371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:11.945540 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:11.946174 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:11.946190 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:11.947786 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:11.949559 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:11.949565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:11.949578 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:11.949827 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:11.950140 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:11.950143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:11.950145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:11.950276 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:11.950666 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:11.950674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:11.950911 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 00:36:11.951082 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:36:11.951400 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260927 00:36:11.953375 24225 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:11.953625 24225 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:36:11.954171 24225 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:11.955695 24034 ts_manager.cc:194] Registered new tserver with Master: 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659)
I20260927 00:36:11.956763 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:54811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:36:11.968874 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:12.073805 24230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:12.074025 24230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:12.074064 24230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:12.077944 24230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:12.077994 24230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:12.078097 24230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:12.083051 24230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:12.084020 24230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:12.084972 24230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:12.094305 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.094352 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.094352 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.094558 24230 server_base.cc:1034] running on GCE node
I20260927 00:36:12.094931 24230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:12.095506 24230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:12.096702 24230 hybrid_clock.cc:648] HybridClock initialized: now 1790469372096686 us; error 26 us; skew 500 ppm
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:12.099289 24230 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:12.100051 24230 webserver.cc:460] Webserver started at http://127.10.11.194:45387/ using document root <none> and password file <none>
I20260927 00:36:12.100520 24230 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:12.100575 24230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:12.100734 24230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:12.102627 24230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "d90e467e16cd8c1ed414374e9e253ff5"
server_key_iv: "67c5fc3af3c385c469055683b75529a8"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.103111 24230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "d90e467e16cd8c1ed414374e9e253ff5"
server_key_iv: "67c5fc3af3c385c469055683b75529a8"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.107031 24230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:36:12.109181 24245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.110363 24230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:12.110451 24230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "d90e467e16cd8c1ed414374e9e253ff5"
server_key_iv: "67c5fc3af3c385c469055683b75529a8"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.110527 24230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:12.119824 24230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:12.122360 24230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:12.122494 24230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:12.122959 24230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:12.123664 24230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:12.123716 24230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.123754 24230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:12.123786 24230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.132172 24230 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:43633
I20260927 00:36:12.132210 24358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:43633 every 8 connection(s)
I20260927 00:36:12.133082 24230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:12.134311 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24230
I20260927 00:36:12.134433 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:36:12.134748 10287 external_mini_cluster.cc:1428] Setting key f3246c543ce7a634fe3e1d64b40f15df
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:36:12.147290 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.134756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:51047 (local address 127.10.11.254:38669)
0927 00:36:12.134884 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.134888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.135885 (+ 997us) server_negotiation.cc:407] Connection header received
0927 00:36:12.137281 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.137284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.137335 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.137411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:12.138647 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.139492 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.140811 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.141286 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.141414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.143740 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:12.143744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:12.143755 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:12.143784 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:12.145213 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.145722 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.145725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.145726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.145804 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.146083 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.146085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.146086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.146357 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:12.146456 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.146937 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.147068 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:36:12.148188 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.135196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:51047)
0927 00:36:12.135700 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.135780 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.137075 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.137568 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.137574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.137897 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:12.138518 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.138533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.139617 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.139619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.140677 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.140687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.141441 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.141445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.141612 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.142234 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:12.142248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:12.143579 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:12.145361 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:12.145369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:12.145385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:12.145634 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:12.145882 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:12.145884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:12.145885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:12.146002 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:12.146466 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:12.146470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:12.146667 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.146867 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.147136 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":146,"threads_started":1}
I20260927 00:36:12.149289 24359 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:12.149519 24359 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:12.149945 24359 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:12.151005 24034 ts_manager.cc:194] Registered new tserver with Master: e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633)
I20260927 00:36:12.151564 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:51047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:36:12.161229 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:12.265975 24364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:12.266194 24364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:12.266233 24364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:12.270233 24364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:12.270323 24364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:12.270430 24364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:12.275288 24364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:12.276124 24364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:12.276997 24364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:12.285998 24370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.286003 24372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.286202 24364 server_base.cc:1034] running on GCE node
W20260927 00:36:12.286029 24369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.286720 24364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:12.287460 24364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:12.288650 24364 hybrid_clock.cc:648] HybridClock initialized: now 1790469372288622 us; error 37 us; skew 500 ppm
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:12.291317 24364 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:12.292066 24364 webserver.cc:460] Webserver started at http://127.10.11.195:33281/ using document root <none> and password file <none>
I20260927 00:36:12.292567 24364 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:12.292652 24364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:12.292867 24364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:12.294726 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "f831a647546144b6be7f2b23ada08806"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "06c26014d19c9e89d92dcb9ab72729c6"
server_key_iv: "6c06bf3bdea4d86f6c03318f32335c79"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.295248 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "f831a647546144b6be7f2b23ada08806"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "06c26014d19c9e89d92dcb9ab72729c6"
server_key_iv: "6c06bf3bdea4d86f6c03318f32335c79"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.298635 24364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:36:12.300787 24379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.301884 24364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:12.302011 24364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "f831a647546144b6be7f2b23ada08806"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "06c26014d19c9e89d92dcb9ab72729c6"
server_key_iv: "6c06bf3bdea4d86f6c03318f32335c79"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.302150 24364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:12.334152 24364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:12.337249 24364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:12.337453 24364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:12.338009 24364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:12.338819 24364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:12.338865 24364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.338934 24364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:12.338971 24364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.346980 24364 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:33815
I20260927 00:36:12.347005 24492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:33815 every 8 connection(s)
I20260927 00:36:12.347781 24364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:12.356839 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24364
I20260927 00:36:12.356938 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:36:12.357250 10287 external_mini_cluster.cc:1428] Setting key 2ce84a3efbb6b4a3f307e1b09d0d03ec
I20260927 00:36:12.360262 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.349418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34117 (local address 127.10.11.254:38669)
0927 00:36:12.349569 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.349574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.350342 (+ 768us) server_negotiation.cc:407] Connection header received
0927 00:36:12.351610 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.351613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.351658 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.351732 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:12.352735 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.353225 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.354309 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.354504 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.354579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.356769 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:12.356772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:12.356774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:12.356796 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:12.358418 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.358976 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.358980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.358981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.359065 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.359423 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.359426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.359427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.359609 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:12.359683 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.360068 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.360154 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:36:12.360929 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.349764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:34117)
0927 00:36:12.350209 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.350253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.351434 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.351885 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.351891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.352209 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:12.352621 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.352633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.353329 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.353331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.354211 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.354220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.354618 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.354620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.354751 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.355246 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:12.355261 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:12.356574 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:12.358533 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:12.358538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:12.358550 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:12.358880 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:12.359151 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:12.359153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:12.359155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:12.359324 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:12.359696 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:12.359700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:12.359886 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.360022 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.360222 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260927 00:36:12.361850 24493 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:12.362015 24493 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:12.362434 24493 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:12.363317 24034 ts_manager.cc:194] Registered new tserver with Master: f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
I20260927 00:36:12.363847 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:34117
I20260927 00:36:12.370846 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:12.379010 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.372594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.10.11.254:38669)
0927 00:36:12.372702 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.372705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.372826 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:36:12.372915 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.372917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.372946 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.373004 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:12.373610 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.374074 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.374896 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.375083 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.375152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.376316 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:12.376318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:12.376320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:12.376340 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:12.377682 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.378029 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.378031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.378032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.378138 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:12.378385 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:12.378387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:12.378388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:12.378571 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:12.378669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.378829 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.378908 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:36:12.381465 24034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:36:12.384315 24034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:36:12.397866 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:42659 (local address 127.0.0.1:51710)
0927 00:36:12.392382 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.392429 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.392550 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.393839 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.393842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.393862 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:12.394087 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.394110 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.395525 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.395528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.396594 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.396601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397373 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.397478 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397511 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.397612 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.397718 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":96,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260927 00:36:12.398051 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.10.11.195:33815)
0927 00:36:12.393433 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.393442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.393460 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:36:12.393517 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.393520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.393730 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.393861 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:12.394289 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.394991 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.396535 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397346 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397583 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.397807 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.397898 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":108,"threads_started":1}
I20260927 00:36:12.398207 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.10.11.193:42659)
0927 00:36:12.393351 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.393363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.393385 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:36:12.393458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.393465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.393668 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.394077 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:12.394270 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.395381 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.396699 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397250 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397374 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397512 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.397950 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.398052 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":126,"threads_started":1}
I20260927 00:36:12.398279 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:43633 (local address 127.0.0.1:57790)
0927 00:36:12.393014 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.393027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.393114 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.393778 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.393786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.393815 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:12.394045 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.394053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.395557 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.395560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.396641 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.396652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397839 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.397924 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397946 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.398017 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.398092 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":49,"thread_start_us":105,"threads_started":1}
I20260927 00:36:12.398625 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33815 (local address 127.0.0.1:59642)
0927 00:36:12.392663 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.392677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.392758 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.394007 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.394009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.394024 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:12.394193 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.394200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.395115 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.395121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.396415 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.396425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397588 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.397675 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397692 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.397760 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.397825 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20260927 00:36:12.398665 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.392932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.10.11.194:43633)
0927 00:36:12.393307 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.393321 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.393345 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:36:12.393419 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.393424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.393619 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.393779 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:12.394197 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.395381 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.396788 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.397552 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.397647 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.397777 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.398409 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.398507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":144,"threads_started":1}
I20260927 00:36:12.401054 24293 tablet_service.cc:1467] Processing CreateTablet for tablet d0f908d6531544618a2fee22a9bcfb5b (DEFAULT_TABLE table=test-table [id=6819f85a37b04b879794cacebee270de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:12.401170 24427 tablet_service.cc:1467] Processing CreateTablet for tablet d0f908d6531544618a2fee22a9bcfb5b (DEFAULT_TABLE table=test-table [id=6819f85a37b04b879794cacebee270de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:12.401168 24159 tablet_service.cc:1467] Processing CreateTablet for tablet d0f908d6531544618a2fee22a9bcfb5b (DEFAULT_TABLE table=test-table [id=6819f85a37b04b879794cacebee270de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:12.402020 24293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f908d6531544618a2fee22a9bcfb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:12.402143 24159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f908d6531544618a2fee22a9bcfb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:12.402149 24427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f908d6531544618a2fee22a9bcfb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:12.408761 24511 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Bootstrap starting.
I20260927 00:36:12.409521 24512 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Bootstrap starting.
I20260927 00:36:12.409722 24510 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Bootstrap starting.
I20260927 00:36:12.411222 24512 tablet_bootstrap.cc:654] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:12.411537 24511 tablet_bootstrap.cc:654] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:12.412292 24510 tablet_bootstrap.cc:654] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:12.412318 24512 log.cc:824] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:12.412441 24511 log.cc:824] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:12.413015 24510 log.cc:824] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:12.414309 24511 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: No bootstrap required, opened a new log
I20260927 00:36:12.414331 24512 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: No bootstrap required, opened a new log
I20260927 00:36:12.414494 24512 ts_tablet_manager.cc:1397] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:36:12.414489 24511 ts_tablet_manager.cc:1397] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:36:12.414938 24510 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: No bootstrap required, opened a new log
I20260927 00:36:12.415100 24510 ts_tablet_manager.cc:1397] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:36:12.417563 24512 raft_consensus.cc:348] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.417834 24512 raft_consensus.cc:374] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:12.417905 24512 raft_consensus.cc:729] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f831a647546144b6be7f2b23ada08806, State: Initialized, Role: FOLLOWER
I20260927 00:36:12.418223 24511 raft_consensus.cc:348] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.418381 24512 consensus_queue.cc:260] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.418466 24511 raft_consensus.cc:374] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:12.418521 24511 raft_consensus.cc:729] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601c11ab81954f2095240a3615e054b6, State: Initialized, Role: FOLLOWER
I20260927 00:36:12.418645 24510 raft_consensus.cc:348] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.418900 24510 raft_consensus.cc:374] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:12.418979 24510 raft_consensus.cc:729] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37493f0c3c341b2a96d17ee5c8d05ed, State: Initialized, Role: FOLLOWER
I20260927 00:36:12.418962 24511 consensus_queue.cc:260] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.419126 24493 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
I20260927 00:36:12.419394 24512 ts_tablet_manager.cc:1428] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:36:12.419386 24510 consensus_queue.cc:260] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.419898 24511 ts_tablet_manager.cc:1428] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:36:12.419968 24359 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
I20260927 00:36:12.420228 24225 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
I20260927 00:36:12.421010 24510 ts_tablet_manager.cc:1428] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 00:36:12.438714 24226 tablet.cc:2369] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:12.547366 24517 raft_consensus.cc:482] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:12.547585 24517 raft_consensus.cc:504] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.548736 24517 leader_election.cc:288] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
I20260927 00:36:12.552510 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.549151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:49717 (local address 127.10.11.195:33815)
0927 00:36:12.549403 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.549408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.549424 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:36:12.549480 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.549483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.549531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.549614 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:12.549890 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.550457 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.551551 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.551947 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.552069 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.552118 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.552317 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.552380 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260927 00:36:12.552557 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.549089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:55053 (local address 127.10.11.194:43633)
0927 00:36:12.549291 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.549296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.549451 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:36:12.549601 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.549604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.549654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.549742 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:12.550009 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.550523 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.551585 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.551999 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.552125 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.552175 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.552371 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.552437 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 00:36:12.552596 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.548949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:43633 (local address 127.10.11.193:55053)
0927 00:36:12.549363 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.549374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.549495 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.549749 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.549752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.549778 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:12.549903 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.549908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.550638 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.550643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.551458 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.551467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.552118 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.552121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.552223 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.552250 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.552356 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.552428 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":89,"thread_start_us":112,"threads_started":1}
I20260927 00:36:12.552582 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.548949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33815 (local address 127.10.11.193:49717)
0927 00:36:12.549275 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.549290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:12.549394 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:12.549634 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:12.549637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:12.549653 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:12.549773 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.549778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.550566 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.550571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.551419 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:12.551429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.552064 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.552067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:12.552163 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.552191 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:36:12.552297 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:36:12.552368 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":38,"thread_start_us":64,"threads_started":1}
I20260927 00:36:12.553316 24447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f908d6531544618a2fee22a9bcfb5b" candidate_uuid: "601c11ab81954f2095240a3615e054b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a647546144b6be7f2b23ada08806" is_pre_election: true
I20260927 00:36:12.553318 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f908d6531544618a2fee22a9bcfb5b" candidate_uuid: "601c11ab81954f2095240a3615e054b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" is_pre_election: true
I20260927 00:36:12.553640 24313 raft_consensus.cc:2454] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601c11ab81954f2095240a3615e054b6 in term 0.
I20260927 00:36:12.553640 24447 raft_consensus.cc:2454] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601c11ab81954f2095240a3615e054b6 in term 0.
I20260927 00:36:12.554201 24113 leader_election.cc:302] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 601c11ab81954f2095240a3615e054b6, e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806; no voters:
I20260927 00:36:12.554459 24517 raft_consensus.cc:2781] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:12.554548 24517 raft_consensus.cc:482] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:12.554577 24517 raft_consensus.cc:3037] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:12.555594 24517 raft_consensus.cc:504] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.555912 24517 leader_election.cc:288] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 1 election: Requested vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
I20260927 00:36:12.556243 24447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f908d6531544618a2fee22a9bcfb5b" candidate_uuid: "601c11ab81954f2095240a3615e054b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a647546144b6be7f2b23ada08806"
I20260927 00:36:12.556252 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f908d6531544618a2fee22a9bcfb5b" candidate_uuid: "601c11ab81954f2095240a3615e054b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
I20260927 00:36:12.556370 24447 raft_consensus.cc:3037] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:12.556370 24313 raft_consensus.cc:3037] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:12.557610 24313 raft_consensus.cc:2454] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601c11ab81954f2095240a3615e054b6 in term 1.
I20260927 00:36:12.557610 24447 raft_consensus.cc:2454] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601c11ab81954f2095240a3615e054b6 in term 1.
I20260927 00:36:12.557942 24115 leader_election.cc:302] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 601c11ab81954f2095240a3615e054b6, e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806; no voters:
I20260927 00:36:12.558135 24517 raft_consensus.cc:2781] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:12.558322 24517 raft_consensus.cc:686] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 LEADER]: Becoming Leader. State: Replica: 601c11ab81954f2095240a3615e054b6, State: Running, Role: LEADER
I20260927 00:36:12.558619 24517 consensus_queue.cc:237] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:12.561573 24033 catalog_manager.cc:5495] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 601c11ab81954f2095240a3615e054b6 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "601c11ab81954f2095240a3615e054b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } health_report { overall_health: UNKNOWN } } }
I20260927 00:36:12.582842 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.579759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.10.11.193:42659)
0927 00:36:12.579875 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:12.579878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:12.579893 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:36:12.580031 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:12.580033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:12.580078 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:12.580160 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:12.580360 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.580817 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.581549 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:12.581746 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:12.581816 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:12.582045 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:12.582491 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:12.582582 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:12.582636 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:36:12.582677 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:36:12.588512 24447 raft_consensus.cc:1261] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Refusing update from remote peer 601c11ab81954f2095240a3615e054b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:12.588512 24313 raft_consensus.cc:1261] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Refusing update from remote peer 601c11ab81954f2095240a3615e054b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:12.589210 24517 consensus_queue.cc:1035] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:12.589422 24521 consensus_queue.cc:1035] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:12.604061 24032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:12.604246 24032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:12.607165 24032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Sending DeleteTablet for 3 replicas of tablet d0f908d6531544618a2fee22a9bcfb5b
I20260927 00:36:12.607993 24159 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f908d6531544618a2fee22a9bcfb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:12 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:51710
I20260927 00:36:12.608117 24293 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f908d6531544618a2fee22a9bcfb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:12 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:57790
I20260927 00:36:12.608232 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24096
I20260927 00:36:12.608443 24427 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f908d6531544618a2fee22a9bcfb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:12 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:59642
I20260927 00:36:12.608886 24536 tablet_replica.cc:330] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: stopping tablet replica
I20260927 00:36:12.609226 24536 raft_consensus.cc:2229] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:12.609565 24536 raft_consensus.cc:2258] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:12.609946 24535 tablet_replica.cc:330] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: stopping tablet replica
I20260927 00:36:12.610260 24535 raft_consensus.cc:2229] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:12.610548 24535 raft_consensus.cc:2258] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:12.611008 24536 ts_tablet_manager.cc:1905] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:12.611876 24535 ts_tablet_manager.cc:1905] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:12.613917 24536 ts_tablet_manager.cc:1918] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:12.614008 24536 log.cc:1196] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/d0f908d6531544618a2fee22a9bcfb5b
I20260927 00:36:12.614382 24536 ts_tablet_manager.cc:1939] T d0f908d6531544618a2fee22a9bcfb5b P f831a647546144b6be7f2b23ada08806: Deleting consensus metadata
I20260927 00:36:12.614648 24535 ts_tablet_manager.cc:1918] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:12.614739 24535 log.cc:1196] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/d0f908d6531544618a2fee22a9bcfb5b
I20260927 00:36:12.615072 24535 ts_tablet_manager.cc:1939] T d0f908d6531544618a2fee22a9bcfb5b P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting consensus metadata
W20260927 00:36:12.615528 24021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:42659 (error 108)
I20260927 00:36:12.616142 24019 catalog_manager.cc:4841] TS f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): tablet d0f908d6531544618a2fee22a9bcfb5b (table test-table [id=6819f85a37b04b879794cacebee270de]) successfully deleted
I20260927 00:36:12.616323 24021 catalog_manager.cc:4841] TS e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): tablet d0f908d6531544618a2fee22a9bcfb5b (table test-table [id=6819f85a37b04b879794cacebee270de]) successfully deleted
I20260927 00:36:12.616380 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24230
I20260927 00:36:12.616724 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:12.616199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:42659 (local address 127.0.0.1:51722)
0927 00:36:12.616545 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:12.616620 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
W20260927 00:36:12.617208 24021 catalog_manager.cc:4584] TS 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0f908d6531544618a2fee22a9bcfb5b: Network error: Client connection negotiation failed: client connection to 127.10.11.193:42659: connect: Connection refused (error 111)
I20260927 00:36:12.622520 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24364
I20260927 00:36:12.627729 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24002
I20260927 00:36:12.633786 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--webserver_interface=127.10.11.254
--webserver_port=45373
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:12.738406 24538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:12.738648 24538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:12.738730 24538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:12.742708 24538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:12.742784 24538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:12.742846 24538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:12.742902 24538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:12.742937 24538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:12.747896 24538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:12.748791 24538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:12.749679 24538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:12.761997 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.761941 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.761930 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.762315 24538 server_base.cc:1034] running on GCE node
I20260927 00:36:12.762908 24538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:12.763978 24538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:12.765161 24538 hybrid_clock.cc:648] HybridClock initialized: now 1790469372765149 us; error 27 us; skew 500 ppm
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:12.767899 24538 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:12.768733 24538 webserver.cc:460] Webserver started at http://127.10.11.254:45373/ using document root <none> and password file <none>
I20260927 00:36:12.769302 24538 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:12.769409 24538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:12.773284 24538 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:12.775652 24554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.776855 24538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:12.776991 24538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "9b99071e7c9241738721192ac327146d"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "baf0c94ebae918dadf4ab010d49ec3b5"
server_key_iv: "4f4b61c9fb04ff5240926c884e93ddcd"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.777573 24538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:12.810225 24538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:12.813022 24538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:12.813220 24538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:12.819957 24538 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:38669
I20260927 00:36:12.820223 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:38669 every 8 connection(s)
I20260927 00:36:12.821028 24538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:12.825255 24607 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:36:12.826876 24607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap starting.
I20260927 00:36:12.829487 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24538
I20260927 00:36:12.829893 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:42659
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=33951
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:12.844141 24607 log.cc:824] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:12.852594 24607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:12.852957 24607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap complete.
I20260927 00:36:12.856915 24607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:12.857378 24607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b99071e7c9241738721192ac327146d, State: Initialized, Role: FOLLOWER
I20260927 00:36:12.857888 24607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:12.858034 24607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:12.858148 24607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:12.858261 24607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:36:12.860060 24607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:12.860388 24607 leader_election.cc:302] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b99071e7c9241738721192ac327146d; no voters:
I20260927 00:36:12.860752 24607 leader_election.cc:288] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:36:12.860829 24611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:36:12.861078 24611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 LEADER]: Becoming Leader. State: Replica: 9b99071e7c9241738721192ac327146d, State: Running, Role: LEADER
I20260927 00:36:12.861423 24611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:12.861972 24607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:12.862772 24612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:12.862891 24612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:12.863269 24613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b99071e7c9241738721192ac327146d. Latest consensus state: current_term: 2 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:12.863402 24613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:12.864117 24620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:12.868826 24620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:12.869429 24620 catalog_manager.cc:1259] Loaded cluster ID: d8e8af573eae4e3ab91c58239dd3496b
I20260927 00:36:12.869490 24620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:12.870898 24620 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:36:12.875834 24620 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:36:12.880908 24620 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:12.881588 24620 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Loaded TSK: 0
I20260927 00:36:12.882747 24620 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:36:12.939169 24609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:12.939419 24609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:12.939488 24609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:12.943401 24609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:12.943472 24609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:12.943552 24609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:12.948597 24609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:42659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=33951
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:12.949515 24609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:12.950491 24609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:12.960533 24634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:12.960529 24635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.960748 24609 server_base.cc:1034] running on GCE node
W20260927 00:36:12.960551 24637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:12.961324 24609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:12.961941 24609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:12.963146 24609 hybrid_clock.cc:648] HybridClock initialized: now 1790469372963101 us; error 45 us; skew 500 ppm
Sep 27 00:36:12 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:12.965930 24609 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:12.966858 24609 webserver.cc:460] Webserver started at http://127.10.11.193:33951/ using document root <none> and password file <none>
I20260927 00:36:12.967374 24609 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:12.967473 24609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:12.971212 24609 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 00:36:12.973276 24644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:12.974529 24609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:12.974617 24609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "601c11ab81954f2095240a3615e054b6"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "2d710ebf1c27025114635b390110b2eb"
server_key_iv: "52660eba66cfacc694e3014d1b421900"
server_key_version: "encryptionkey@0"
I20260927 00:36:12.975121 24609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:13.032675 24609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:13.035523 24609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:13.035719 24609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:13.036363 24609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:13.037390 24651 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:36:13.041349 24609 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:36:13.041404 24609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 00:36:13.041469 24609 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:36:13.043100 24609 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:36:13.043143 24609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:13.043421 24651 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Bootstrap starting.
I20260927 00:36:13.053416 24609 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:42659
I20260927 00:36:13.053494 24758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:42659 every 8 connection(s)
I20260927 00:36:13.054382 24609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:13.056627 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24609
I20260927 00:36:13.057192 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:43633
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=45387
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:13.062038 24651 log.cc:824] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:13.071138 24651 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:13.071523 24651 tablet_bootstrap.cc:492] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Bootstrap complete.
I20260927 00:36:13.071696 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.056463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:49107 (local address 127.10.11.254:38669)
0927 00:36:13.056922 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.056941 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.057454 (+ 513us) server_negotiation.cc:407] Connection header received
0927 00:36:13.058804 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.058820 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.059222 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.059588 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:13.060991 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.062034 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.063714 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.064228 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.064404 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.066807 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:13.066813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:13.066833 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:13.066859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:13.068553 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.069101 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.069107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.069118 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.069229 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.069564 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.069569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.069571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.069882 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:13.070019 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.070561 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.070796 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":118,"threads_started":1}
I20260927 00:36:13.071702 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.056367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:49107)
0927 00:36:13.057278 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.057337 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.058392 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.059726 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.059738 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.060204 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:13.060820 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.060838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.062208 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.062212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.063597 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.063607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.064367 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.064370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.064546 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.065169 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:13.065185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:13.066626 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:13.068715 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.068720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.068738 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.068982 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.069332 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.069336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.069338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.069470 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.069993 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:13.069999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:13.070275 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.070458 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.070773 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":131,"threads_started":1}
I20260927 00:36:13.072924 24651 ts_tablet_manager.cc:1397] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.012s
I20260927 00:36:13.074035 24759 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:13.074277 24759 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:13.074702 24759 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:13.075644 24651 raft_consensus.cc:348] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:13.076143 24651 raft_consensus.cc:729] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601c11ab81954f2095240a3615e054b6, State: Initialized, Role: FOLLOWER
I20260927 00:36:13.076597 24651 consensus_queue.cc:260] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } } peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } }
I20260927 00:36:13.077046 24571 ts_manager.cc:194] Registered new tserver with Master: 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659)
I20260927 00:36:13.077499 24651 ts_tablet_manager.cc:1428] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260927 00:36:13.080482 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:49107
I20260927 00:36:13.082858 24759 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:36:13.090282 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.079816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:42659 (local address 127.0.0.1:51730)
0927 00:36:13.080524 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.080570 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.080744 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.081324 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.081330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.081716 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:13.081845 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.081853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.083151 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.083154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.084105 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.084119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.084680 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.084684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.084793 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.085290 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:13.085304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:13.086823 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:13.088627 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.088634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.088639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.088862 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.089145 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.089149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.089151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.089238 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.089661 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:13.089667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:13.089787 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.089921 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.090090 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":87,"threads_started":1}
I20260927 00:36:13.090313 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.079959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.10.11.193:42659)
0927 00:36:13.080256 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.080267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.080651 (+ 384us) server_negotiation.cc:407] Connection header received
0927 00:36:13.080913 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.080918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.081120 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.081261 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:13.081967 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.082977 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.084225 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.084569 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.084648 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.086953 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:13.086969 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:13.086977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:13.086997 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:13.088505 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.088970 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.088982 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.088984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.089053 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.089333 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.089336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.089337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.089548 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:13.089684 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.089993 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.090161 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260927 00:36:13.091290 24693 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f908d6531544618a2fee22a9bcfb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:12 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:51730
I20260927 00:36:13.092000 24768 tablet_replica.cc:330] stopping tablet replica
I20260927 00:36:13.092267 24768 raft_consensus.cc:2229] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:13.092485 24768 raft_consensus.cc:2258] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:13.093323 24768 ts_tablet_manager.cc:1905] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:13.096423 24768 ts_tablet_manager.cc:1918] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:13.096498 24768 log.cc:1196] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/d0f908d6531544618a2fee22a9bcfb5b
I20260927 00:36:13.096758 24768 ts_tablet_manager.cc:1939] T d0f908d6531544618a2fee22a9bcfb5b P 601c11ab81954f2095240a3615e054b6: Deleting consensus metadata
I20260927 00:36:13.098227 24558 catalog_manager.cc:4841] TS 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659): tablet d0f908d6531544618a2fee22a9bcfb5b (table test-table [id=6819f85a37b04b879794cacebee270de]) successfully deleted
W20260927 00:36:13.173656 24763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:13.173990 24763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:13.174062 24763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:13.180186 24763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:13.180259 24763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:13.180353 24763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:13.185432 24763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:43633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=45387
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:13.186393 24763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:13.187294 24763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:13.196069 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:13.196075 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:13.196075 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:13.196163 24763 server_base.cc:1034] running on GCE node
I20260927 00:36:13.196668 24763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:13.197224 24763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:13.198407 24763 hybrid_clock.cc:648] HybridClock initialized: now 1790469373198394 us; error 26 us; skew 500 ppm
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:13.200980 24763 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:13.201687 24763 webserver.cc:460] Webserver started at http://127.10.11.194:45387/ using document root <none> and password file <none>
I20260927 00:36:13.202229 24763 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:13.202303 24763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:13.205700 24763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:13.207535 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.208639 24763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:13.208758 24763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "d90e467e16cd8c1ed414374e9e253ff5"
server_key_iv: "67c5fc3af3c385c469055683b75529a8"
server_key_version: "encryptionkey@0"
I20260927 00:36:13.209221 24763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:13.239586 24763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:13.242780 24763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:13.242982 24763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:13.243553 24763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:13.244320 24763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:13.244365 24763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.244434 24763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:13.244469 24763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.252671 24763 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:43633
I20260927 00:36:13.252719 24896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:43633 every 8 connection(s)
I20260927 00:36:13.253585 24763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:13.255124 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24763
I20260927 00:36:13.255595 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:33815
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=33281
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:13.267696 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.255301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:51593 (local address 127.10.11.254:38669)
0927 00:36:13.255432 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.255442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.256414 (+ 972us) server_negotiation.cc:407] Connection header received
0927 00:36:13.258363 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.258366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.258435 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.258530 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:13.259637 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.260173 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.261323 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.261573 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.261706 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.264192 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:13.264195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:13.264198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:13.264228 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:13.265671 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.266217 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.266221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.266223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.266306 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.266648 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.266659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.266660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.266829 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:13.266934 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.267395 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.267519 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:36:13.268412 24899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.255766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:51593)
0927 00:36:13.256258 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.256296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.258042 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.258689 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.258696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.259016 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:13.259479 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.259492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.260323 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.260332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.261193 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.261202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.261743 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.261747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.261912 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.262498 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:13.262513 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:13.264026 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:13.265804 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.265809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.265821 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.266067 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.266407 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.266409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.266411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.266542 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.266932 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:13.266939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:13.267168 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.267339 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.267630 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260927 00:36:13.269538 24897 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:13.269801 24897 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:13.270325 24897 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:13.271436 24571 ts_manager.cc:194] Registered new tserver with Master: e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633)
I20260927 00:36:13.272187 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:51593
W20260927 00:36:13.363001 24900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:13.363293 24900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:13.363366 24900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:13.367484 24900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:13.367534 24900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:13.367619 24900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:13.372690 24900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=33281
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:13.373585 24900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:13.374543 24900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:13.383975 24905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:13.384049 24900 server_base.cc:1034] running on GCE node
W20260927 00:36:13.383982 24906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:13.383975 24908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:13.384656 24900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:13.385160 24900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:13.386315 24900 hybrid_clock.cc:648] HybridClock initialized: now 1790469373386303 us; error 25 us; skew 500 ppm
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:13.388885 24900 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:13.389640 24900 webserver.cc:460] Webserver started at http://127.10.11.195:33281/ using document root <none> and password file <none>
I20260927 00:36:13.390202 24900 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:13.390303 24900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:13.393692 24900 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:13.395635 24915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.396709 24900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:13.396842 24900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "f831a647546144b6be7f2b23ada08806"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "06c26014d19c9e89d92dcb9ab72729c6"
server_key_iv: "6c06bf3bdea4d86f6c03318f32335c79"
server_key_version: "encryptionkey@0"
I20260927 00:36:13.397331 24900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:13.413898 24900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:13.416934 24900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:13.417073 24900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:13.417520 24900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:13.418274 24900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:13.418322 24900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.418360 24900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:13.418380 24900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.426272 24900 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:33815
I20260927 00:36:13.426304 25028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:33815 every 8 connection(s)
I20260927 00:36:13.427175 24900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:13.431876 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 24900
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:13.440521 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.432489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.10.11.254:38669)
0927 00:36:13.432704 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.432708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.432850 (+ 142us) server_negotiation.cc:407] Connection header received
0927 00:36:13.432950 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.432953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.432995 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.433078 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:13.433968 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.434485 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.435598 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.435813 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.435887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.437245 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:13.437249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:13.437252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:13.437275 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:13.438828 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.439290 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.439296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.439298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.439382 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.439637 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.439640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.439642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.439859 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:13.440053 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.440239 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.440336 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260927 00:36:13.441198 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.428819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:48149 (local address 127.10.11.254:38669)
0927 00:36:13.428954 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.428957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.429684 (+ 727us) server_negotiation.cc:407] Connection header received
0927 00:36:13.430988 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.430992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.431048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.431155 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:13.432267 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.432996 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.434529 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.434789 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.434862 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.437405 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:13.437408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:13.437410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:13.437443 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:13.439063 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.439673 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.439675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.439676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.439748 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:13.440103 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:13.440106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:13.440110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:13.440355 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:13.440448 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.441010 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.441101 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:36:13.442327 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.429156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:48149)
0927 00:36:13.429548 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.429585 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.430786 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.431329 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.431335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.431645 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:13.432125 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.432139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.433121 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.433125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.434410 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.434420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.434929 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.434934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.435110 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.435657 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:13.435672 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:13.437231 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:13.439179 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.439183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.439192 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.439557 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.439839 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:13.439842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:13.439843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:13.440005 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:13.440509 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:13.440513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:13.440795 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.440987 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.441509 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260927 00:36:13.443225 25029 heartbeater.cc:347] Connected to a master server at 127.10.11.254:38669
I20260927 00:36:13.443456 25029 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:13.443885 25029 heartbeater.cc:511] Master 127.10.11.254:38669 requested a full tablet report, sending...
I20260927 00:36:13.444804 24571 ts_manager.cc:194] Registered new tserver with Master: f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
I20260927 00:36:13.445322 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:48149
I20260927 00:36:13.453712 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:13.457369 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.454310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.10.11.254:38669)
0927 00:36:13.454435 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.454439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.454623 (+ 184us) server_negotiation.cc:407] Connection header received
0927 00:36:13.454719 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.454721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.454796 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.454871 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:13.455048 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.455514 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.456253 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.456455 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.456523 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.456766 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:13.457060 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:13.457141 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.457200 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.457235 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:36:13.457962 24571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:36:13.461689 24571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:36:13.469304 24693 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd (DEFAULT_TABLE table=test-table [id=71384af74afa434495df7b0bc64389a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:13.469910 24693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2cf6a988024dfb9aa0fa31600eccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:13.472617 24651 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Bootstrap starting.
I20260927 00:36:13.473582 24651 tablet_bootstrap.cc:654] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:13.474151 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.468614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.10.11.194:43633)
0927 00:36:13.469096 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.469108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.469131 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:36:13.469218 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.469226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.469434 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.469630 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:13.469898 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.471073 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.472627 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.473454 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.473567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.473696 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.473880 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.473981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":120,"threads_started":1}
I20260927 00:36:13.474076 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.468255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:43633 (local address 127.0.0.1:57806)
0927 00:36:13.468365 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.468381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.468457 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.469621 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.469626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.469649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:13.469738 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.469743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.471213 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.471217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.472468 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.472479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.473585 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.473588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.473685 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.473735 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.473838 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.473934 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:36:13.474681 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.468528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33815 (local address 127.0.0.1:59656)
0927 00:36:13.468795 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.468808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.468890 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.469872 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.469874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.469889 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:13.469968 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.469974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.471586 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.471591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.472434 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.472443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.474056 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.474060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.474181 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.474207 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.474262 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.474330 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260927 00:36:13.474834 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.468914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.10.11.195:33815)
0927 00:36:13.469219 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.469229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.469247 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:36:13.469304 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.469308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.469440 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.469629 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:13.470066 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.471416 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.472555 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.473926 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.474415 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.474551 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.474603 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.474713 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260927 00:36:13.475309 24651 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: No bootstrap required, opened a new log
I20260927 00:36:13.475411 24651 ts_tablet_manager.cc:1397] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:36:13.475837 24651 raft_consensus.cc:348] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.475951 24651 raft_consensus.cc:374] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:13.475991 24651 raft_consensus.cc:729] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601c11ab81954f2095240a3615e054b6, State: Initialized, Role: FOLLOWER
I20260927 00:36:13.476104 24651 consensus_queue.cc:260] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.476815 24651 ts_tablet_manager.cc:1428] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:36:13.477818 24963 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd (DEFAULT_TABLE table=test-table [id=71384af74afa434495df7b0bc64389a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:13.478911 24963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2cf6a988024dfb9aa0fa31600eccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:13.478947 24831 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd (DEFAULT_TABLE table=test-table [id=71384af74afa434495df7b0bc64389a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:13.479939 24831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2cf6a988024dfb9aa0fa31600eccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:13.486734 25042 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Bootstrap starting.
I20260927 00:36:13.487300 25043 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Bootstrap starting.
I20260927 00:36:13.489768 25043 tablet_bootstrap.cc:654] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:13.489768 25042 tablet_bootstrap.cc:654] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:13.490684 25042 log.cc:824] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:13.490675 25043 log.cc:824] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:13.492363 25042 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: No bootstrap required, opened a new log
I20260927 00:36:13.492363 25043 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: No bootstrap required, opened a new log
I20260927 00:36:13.492558 25043 ts_tablet_manager.cc:1397] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:36:13.492558 25042 ts_tablet_manager.cc:1397] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:36:13.495301 25042 raft_consensus.cc:348] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.495309 25043 raft_consensus.cc:348] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.495496 25042 raft_consensus.cc:374] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:13.495527 25043 raft_consensus.cc:374] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:13.495560 25042 raft_consensus.cc:729] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f831a647546144b6be7f2b23ada08806, State: Initialized, Role: FOLLOWER
I20260927 00:36:13.495596 25043 raft_consensus.cc:729] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37493f0c3c341b2a96d17ee5c8d05ed, State: Initialized, Role: FOLLOWER
I20260927 00:36:13.495988 25043 consensus_queue.cc:260] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.495985 25042 consensus_queue.cc:260] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.496596 24897 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
I20260927 00:36:13.496965 25042 ts_tablet_manager.cc:1428] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 00:36:13.497138 25043 ts_tablet_manager.cc:1428] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:36:13.497233 25029 heartbeater.cc:503] Master 127.10.11.254:38669 was elected leader, sending a full tablet report...
W20260927 00:36:13.504587 24898 tablet.cc:2369] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:13.555500 24760 tablet.cc:2369] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:36:13.678258 25030 tablet.cc:2369] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:13.716943 25046 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:13.717144 25046 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.718271 25046 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815), 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659)
I20260927 00:36:13.721817 24899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.718474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33815 (local address 127.10.11.194:55731)
0927 00:36:13.718634 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.718651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.718749 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.719037 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.719040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.719057 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:13.719191 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.719197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.719941 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.719944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.720751 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.720760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.721393 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.721396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.721502 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.721525 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.721612 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.721684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":23,"spinlock_wait_cycles":27520}
I20260927 00:36:13.721813 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.718612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:55731 (local address 127.10.11.195:33815)
0927 00:36:13.718777 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.718783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.718802 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:36:13.718862 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.718865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.718915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.719008 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:13.719311 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.719825 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.720873 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.721273 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.721399 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.721448 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.721626 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.721689 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:36:13.722195 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.718662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:51145 (local address 127.10.11.193:42659)
0927 00:36:13.718926 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.718931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.719074 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:36:13.719239 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.719247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.719302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.719413 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:13.719686 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.720183 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.721218 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.721663 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.721792 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.721940 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.721979 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.722041 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260927 00:36:13.722209 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.718512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:42659 (local address 127.10.11.194:51145)
0927 00:36:13.718971 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.718981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:13.719119 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:13.719422 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:13.719425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:13.719440 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:13.719564 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.719569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.720297 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.720302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.721095 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:13.721105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.721786 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.721789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:13.721879 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.721900 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:36:13.721963 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:36:13.722027 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":155,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260927 00:36:13.722728 24983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2cf6a988024dfb9aa0fa31600eccdd" candidate_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a647546144b6be7f2b23ada08806" is_pre_election: true
I20260927 00:36:13.722709 24713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2cf6a988024dfb9aa0fa31600eccdd" candidate_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601c11ab81954f2095240a3615e054b6" is_pre_election: true
I20260927 00:36:13.722992 24713 raft_consensus.cc:2454] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37493f0c3c341b2a96d17ee5c8d05ed in term 0.
I20260927 00:36:13.722992 24983 raft_consensus.cc:2454] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37493f0c3c341b2a96d17ee5c8d05ed in term 0.
I20260927 00:36:13.723459 24787 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 601c11ab81954f2095240a3615e054b6, e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806; no voters:
I20260927 00:36:13.723762 25046 raft_consensus.cc:2781] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:13.723842 25046 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:13.723872 25046 raft_consensus.cc:3037] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:13.724910 25046 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.725241 25046 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [CANDIDATE]: Term 1 election: Requested vote from peers f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815), 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659)
I20260927 00:36:13.725600 24713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2cf6a988024dfb9aa0fa31600eccdd" candidate_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601c11ab81954f2095240a3615e054b6"
I20260927 00:36:13.725628 24983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2cf6a988024dfb9aa0fa31600eccdd" candidate_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a647546144b6be7f2b23ada08806"
I20260927 00:36:13.725713 24713 raft_consensus.cc:3037] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:13.725750 24983 raft_consensus.cc:3037] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:13.726742 24713 raft_consensus.cc:2454] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e37493f0c3c341b2a96d17ee5c8d05ed in term 1.
I20260927 00:36:13.726785 24983 raft_consensus.cc:2454] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e37493f0c3c341b2a96d17ee5c8d05ed in term 1.
I20260927 00:36:13.727034 24787 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601c11ab81954f2095240a3615e054b6, e37493f0c3c341b2a96d17ee5c8d05ed; no voters:
I20260927 00:36:13.727196 25046 raft_consensus.cc:2781] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:13.727391 25046 raft_consensus.cc:686] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 LEADER]: Becoming Leader. State: Replica: e37493f0c3c341b2a96d17ee5c8d05ed, State: Running, Role: LEADER
I20260927 00:36:13.727731 25046 consensus_queue.cc:237] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:13.730670 24571 catalog_manager.cc:5495] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed reported cstate change: term changed from 0 to 1, leader changed from <none> to e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194). New cstate: current_term: 1 leader_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } health_report { overall_health: UNKNOWN } } }
I20260927 00:36:13.765096 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.761314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.10.11.194:43633)
0927 00:36:13.761472 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:13.761475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:13.761492 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:36:13.761628 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:13.761631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:13.761703 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:13.761800 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:13.762076 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.762668 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.763666 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:13.763892 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:13.764046 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:13.764333 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:13.764739 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:13.764834 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:13.764939 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:36:13.764990 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:36:13.770984 24983 raft_consensus.cc:1261] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Refusing update from remote peer e37493f0c3c341b2a96d17ee5c8d05ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:13.770987 24713 raft_consensus.cc:1261] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Refusing update from remote peer e37493f0c3c341b2a96d17ee5c8d05ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:13.771687 25050 consensus_queue.cc:1035] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:13.771930 25046 consensus_queue.cc:1035] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:13.778789 25053 mvcc.cc:205] Tried to move back new op lower bound from 7333762554958786560 to 7333762554791231488. Current Snapshot: MvccSnapshot[applied={T|T < 7333762554958786560}]
I20260927 00:36:13.779392 25055 mvcc.cc:205] Tried to move back new op lower bound from 7333762554958786560 to 7333762554791231488. Current Snapshot: MvccSnapshot[applied={T|T < 7333762554958786560}]
I20260927 00:36:13.786639 24571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:13.786798 24571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:13.789150 24571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Sending DeleteTablet for 3 replicas of tablet 0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:13.789849 24693 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:13 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:51730
I20260927 00:36:13.789815 24963 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:13 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:59656
I20260927 00:36:13.789948 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24609
I20260927 00:36:13.790637 24831 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2cf6a988024dfb9aa0fa31600eccdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:13 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:57806
I20260927 00:36:13.791071 25064 tablet_replica.cc:330] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: stopping tablet replica
I20260927 00:36:13.791409 25064 raft_consensus.cc:2229] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:36:13.791699 25063 tablet_replica.cc:330] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: stopping tablet replica
I20260927 00:36:13.791909 25064 raft_consensus.cc:2258] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:13.791970 25063 raft_consensus.cc:2229] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:13.792259 25063 raft_consensus.cc:2258] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:13.793725 25064 ts_tablet_manager.cc:1905] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:13.793763 25063 ts_tablet_manager.cc:1905] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:13.796401 25064 ts_tablet_manager.cc:1918] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:13.796495 25064 log.cc:1196] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:13.796837 25064 ts_tablet_manager.cc:1939] T 0f2cf6a988024dfb9aa0fa31600eccdd P e37493f0c3c341b2a96d17ee5c8d05ed: Deleting consensus metadata
I20260927 00:36:13.797274 25063 ts_tablet_manager.cc:1918] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:13.797437 25063 log.cc:1196] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:13.797833 25063 ts_tablet_manager.cc:1939] T 0f2cf6a988024dfb9aa0fa31600eccdd P f831a647546144b6be7f2b23ada08806: Deleting consensus metadata
I20260927 00:36:13.797884 24558 catalog_manager.cc:4841] TS e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): tablet 0f2cf6a988024dfb9aa0fa31600eccdd (table test-table [id=71384af74afa434495df7b0bc64389a8]) successfully deleted
W20260927 00:36:13.798380 24558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:42659 (error 108)
I20260927 00:36:13.798859 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24763
I20260927 00:36:13.800606 24556 catalog_manager.cc:4841] TS f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): tablet 0f2cf6a988024dfb9aa0fa31600eccdd (table test-table [id=71384af74afa434495df7b0bc64389a8]) successfully deleted
I20260927 00:36:13.801178 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:13.800939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:42659 (local address 127.0.0.1:51736)
0927 00:36:13.801056 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:13.801114 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:36:13.801376 24558 catalog_manager.cc:4584] TS 601c11ab81954f2095240a3615e054b6 (127.10.11.193:42659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f2cf6a988024dfb9aa0fa31600eccdd: Network error: Client connection negotiation failed: client connection to 127.10.11.193:42659: connect: Connection refused (error 111)
I20260927 00:36:13.804869 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24900
I20260927 00:36:13.810237 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 24538
I20260927 00:36:13.816857 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--webserver_interface=127.10.11.254
--webserver_port=45373
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:13.924525 25066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:13.924764 25066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:13.924803 25066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:13.928733 25066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:13.928783 25066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:13.928802 25066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:13.928818 25066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:13.928874 25066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:13.934044 25066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:13.935002 25066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:13.935972 25066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:13.944100 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:13.944100 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:13.944108 25071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:13.944334 25066 server_base.cc:1034] running on GCE node
I20260927 00:36:13.944945 25066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:13.945950 25066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:13.947113 25066 hybrid_clock.cc:648] HybridClock initialized: now 1790469373947077 us; error 46 us; skew 500 ppm
Sep 27 00:36:13 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:13.949891 25066 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:13.950757 25066 webserver.cc:460] Webserver started at http://127.10.11.254:45373/ using document root <none> and password file <none>
I20260927 00:36:13.951278 25066 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:13.951339 25066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:13.954952 25066 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:13.956993 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:13.958281 25066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:13.958415 25066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "9b99071e7c9241738721192ac327146d"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "baf0c94ebae918dadf4ab010d49ec3b5"
server_key_iv: "4f4b61c9fb04ff5240926c884e93ddcd"
server_key_version: "encryptionkey@0"
I20260927 00:36:13.958931 25066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:13.979943 25066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:13.982968 25066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:13.983168 25066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:13.989755 25066 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:38669
I20260927 00:36:13.989828 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:38669 every 8 connection(s)
I20260927 00:36:13.990775 25066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:13.992801 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25066
I20260927 00:36:13.993251 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:42659
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=33951
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:13.995306 25134 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:36:13.997556 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap starting.
I20260927 00:36:14.021034 25134 log.cc:824] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:14.034636 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:14.035054 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d: Bootstrap complete.
I20260927 00:36:14.040860 25134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:14.041363 25134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b99071e7c9241738721192ac327146d, State: Initialized, Role: FOLLOWER
I20260927 00:36:14.041950 25134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:14.042124 25134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:14.042200 25134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:14.042297 25134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:36:14.044651 25134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:14.045015 25134 leader_election.cc:302] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b99071e7c9241738721192ac327146d; no voters:
I20260927 00:36:14.045358 25134 leader_election.cc:288] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:36:14.045472 25138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:36:14.045766 25138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [term 3 LEADER]: Becoming Leader. State: Replica: 9b99071e7c9241738721192ac327146d, State: Running, Role: LEADER
I20260927 00:36:14.046053 25138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } }
I20260927 00:36:14.046583 25134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:14.047534 25139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:14.047663 25139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:14.047513 25140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b99071e7c9241738721192ac327146d. Latest consensus state: current_term: 3 leader_uuid: "9b99071e7c9241738721192ac327146d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99071e7c9241738721192ac327146d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38669 } } }
I20260927 00:36:14.047751 25140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99071e7c9241738721192ac327146d [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:14.049055 25147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:14.053845 25147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:14.054502 25147 catalog_manager.cc:1259] Loaded cluster ID: d8e8af573eae4e3ab91c58239dd3496b
I20260927 00:36:14.054560 25147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:36:14.055466 25147 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4a6909d9ad google::LogMessage::Fail() at ??:0
@ 0x7f4a690a088a google::LogMessage::SendToLog() at ??:0
@ 0x7f4a6909d4de google::LogMessage::Flush() at ??:0
@ 0x7f4a6909ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a739e0fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4a739e1d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4a739de579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4a73a22640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a6b2ef3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4a6b2efc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4a6b2f13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a6b2e0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4a6c9246db start_thread at ??:0
@ 0x7f4a67b9b61f clone at ??:0
W20260927 00:36:14.132141 25136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:14.132361 25136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:14.132401 25136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:14.136381 25136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:14.136430 25136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:14.136514 25136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:14.141645 25136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:42659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=33951
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:14.142652 25136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:14.143591 25136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:14.153191 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.153173 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.153162 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:14.153528 25136 server_base.cc:1034] running on GCE node
I20260927 00:36:14.154009 25136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:14.154858 25136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:14.156068 25136 hybrid_clock.cc:648] HybridClock initialized: now 1790469374156044 us; error 33 us; skew 500 ppm
Sep 27 00:36:14 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:14.159339 25136 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:14.160207 25136 webserver.cc:460] Webserver started at http://127.10.11.193:33951/ using document root <none> and password file <none>
I20260927 00:36:14.160732 25136 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:14.160792 25136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:14.164428 25136 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:14.166690 25185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.167986 25136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:14.168133 25136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "601c11ab81954f2095240a3615e054b6"
format_stamp: "Formatted at 2026-09-27 00:36:11 on dist-test-slave-6gxc"
server_key: "2d710ebf1c27025114635b390110b2eb"
server_key_iv: "52660eba66cfacc694e3014d1b421900"
server_key_version: "encryptionkey@0"
I20260927 00:36:14.168707 25136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:14.185146 25136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:14.188444 25136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:14.188640 25136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:14.189164 25136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:14.190205 25192 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:36:14.197108 25136 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:36:14.197206 25136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260927 00:36:14.197268 25136 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:36:14.199559 25136 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:36:14.199623 25136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:36:14.200124 25192 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Bootstrap starting.
I20260927 00:36:14.210780 25136 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:42659
I20260927 00:36:14.211575 25136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:14.212217 25299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:42659 every 8 connection(s)
I20260927 00:36:14.212545 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25136
I20260927 00:36:14.212977 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:43633
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=45387
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:14.223527 25192 log.cc:824] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:14.232375 25192 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:14.232805 25192 tablet_bootstrap.cc:492] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Bootstrap complete.
I20260927 00:36:14.234251 25192 ts_tablet_manager.cc:1397] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.018s
I20260927 00:36:14.237267 25192 raft_consensus.cc:348] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:14.237798 25192 raft_consensus.cc:729] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601c11ab81954f2095240a3615e054b6, State: Initialized, Role: FOLLOWER
I20260927 00:36:14.238320 25192 consensus_queue.cc:260] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:14.239276 25192 ts_tablet_manager.cc:1428] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 00:36:14.313733 25303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:36:14.215551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:39673)
0927 00:36:14.218440 (+ 2889us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.218502 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:14.219755 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:14.312799 (+ 93044us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2641,"thread_start_us":153,"threads_started":1}
W20260927 00:36:14.314275 25300 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:38669 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:38669: Client connection negotiation failed: client connection to 127.10.11.254:38669: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:36:14.314973 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.314709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:60893)
0927 00:36:14.314843 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.314911 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:36:14.315650 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.315434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:33963)
0927 00:36:14.315543 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.315594 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:36:14.315824 25300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:36:14.323642 25302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:14.323850 25302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:14.323889 25302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:14.327771 25302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:14.327817 25302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:14.327889 25302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:14.332811 25302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:43633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=45387
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:14.333650 25302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:14.334587 25302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:14.343580 25311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.343534 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.343555 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:14.343809 25302 server_base.cc:1034] running on GCE node
I20260927 00:36:14.344223 25302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:14.344777 25302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:14.345955 25302 hybrid_clock.cc:648] HybridClock initialized: now 1790469374345919 us; error 50 us; skew 500 ppm
Sep 27 00:36:14 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:14.348570 25302 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:14.349438 25302 webserver.cc:460] Webserver started at http://127.10.11.194:45387/ using document root <none> and password file <none>
I20260927 00:36:14.349938 25302 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:14.350024 25302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:14.353472 25302 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:14.355880 25320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.357180 25302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:36:14.357276 25302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "e37493f0c3c341b2a96d17ee5c8d05ed"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "d90e467e16cd8c1ed414374e9e253ff5"
server_key_iv: "67c5fc3af3c385c469055683b75529a8"
server_key_version: "encryptionkey@0"
I20260927 00:36:14.357745 25302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:14.379455 25302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:14.382135 25302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:14.382277 25302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:14.382766 25302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:14.383744 25302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:14.383793 25302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.383831 25302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:14.383846 25302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.393079 25302 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:43633
I20260927 00:36:14.393100 25433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:43633 every 8 connection(s)
I20260927 00:36:14.394001 25302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:14.397709 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.396013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:53007)
0927 00:36:14.396495 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.396604 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
W20260927 00:36:14.398212 25434 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:38669 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:38669: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
I20260927 00:36:14.398752 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.398544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:51207)
0927 00:36:14.398646 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.398699 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:36:14.399312 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.399126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:33049)
0927 00:36:14.399222 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.399264 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:36:14.399482 25434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:36:14.400048 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25302
I20260927 00:36:14.400470 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:33815
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:38669
--webserver_port=33281
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:14.504014 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:14.504267 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:14.504338 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:14.508345 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:14.508410 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:14.508492 25437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:14.513336 25437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=33281
--tserver_master_addrs=127.10.11.254:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:14.514236 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:14.515152 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:14.524183 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.524194 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:14.524232 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:14.524272 25437 server_base.cc:1034] running on GCE node
I20260927 00:36:14.524806 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:14.525395 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:14.526571 25437 hybrid_clock.cc:648] HybridClock initialized: now 1790469374526555 us; error 35 us; skew 500 ppm
Sep 27 00:36:14 dist-test-slave-6gxc krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:14.529155 25437 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:14.529958 25437 webserver.cc:460] Webserver started at http://127.10.11.195:33281/ using document root <none> and password file <none>
I20260927 00:36:14.530515 25437 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:14.530603 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:14.534036 25437 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:14.535962 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.537092 25437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:36:14.537212 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "f831a647546144b6be7f2b23ada08806"
format_stamp: "Formatted at 2026-09-27 00:36:12 on dist-test-slave-6gxc"
server_key: "06c26014d19c9e89d92dcb9ab72729c6"
server_key_iv: "6c06bf3bdea4d86f6c03318f32335c79"
server_key_version: "encryptionkey@0"
I20260927 00:36:14.537710 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:14.550601 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:14.553475 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:14.553650 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:14.554205 25437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:14.554932 25437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:14.554999 25437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.555063 25437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:14.555109 25437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:14.563752 25437 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:33815
I20260927 00:36:14.563776 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:33815 every 8 connection(s)
I20260927 00:36:14.564633 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:14.566136 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25437
I20260927 00:36:14.568348 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.566593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:59525)
0927 00:36:14.567110 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.567283 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
W20260927 00:36:14.568832 25566 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:38669 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:38669: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
I20260927 00:36:14.569315 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.569115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:46365)
0927 00:36:14.569222 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.569265 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:36:14.569876 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:14.569693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:51335)
0927 00:36:14.569789 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:14.569829 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:36:14.570057 25566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:36:15.316759 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.316327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:42153)
0927 00:36:15.316602 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:15.316655 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":70,"threads_started":1}
I20260927 00:36:15.400457 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.400072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:47511)
0927 00:36:15.400315 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:15.400374 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260927 00:36:15.549153 25572 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:15.549420 25572 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:15.550609 25572 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
Sep 27 00:36:15 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.195@KRBTEST.COM
Sep 27 00:36:15 dist-test-slave-6gxc krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.194@KRBTEST.COM
I20260927 00:36:15.564412 25575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.551063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:51117 (local address 127.10.11.195:33815)
0927 00:36:15.551432 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:15.551444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:15.551491 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:36:15.552139 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:15.552144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:15.552497 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:15.552758 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:15.554031 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.555080 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556298 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.556781 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556949 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:15.559731 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:15.559736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:15.559746 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:15.559772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:15.561938 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562541 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:15.562549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:15.562554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:15.562681 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:15.563036 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:15.563040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:15.563042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:15.563402 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:15.563560 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:15.564076 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 00:36:15.564281 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":144,"threads_started":1}
I20260927 00:36:15.564431 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.550846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33815 (local address 127.10.11.193:51117)
0927 00:36:15.550992 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:15.551006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:15.551125 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:15.552840 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:15.552846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:15.553373 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:15.553845 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:15.553860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.555233 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.555236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:15.556112 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:15.556121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.556911 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:15.557020 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:15.557648 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:15.557664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:15.559504 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:15.562043 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:15.562081 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:15.562432 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:15.562777 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:15.562786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:15.562937 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:15.563531 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:15.563537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:15.563792 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 00:36:15.563983 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 00:36:15.564242 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":100,"spinlock_wait_cycles":24064}
I20260927 00:36:15.565409 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.550846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:43633 (local address 127.10.11.193:35795)
0927 00:36:15.551360 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:15.551375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:15.551488 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:15.553134 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:15.553136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:15.553370 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:15.553845 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:15.553860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.555204 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.555210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:15.556114 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:15.556123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.556844 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:15.557014 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:15.557699 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:15.557712 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:15.559943 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:15.562162 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:15.562167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:15.562459 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:15.562822 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:15.562826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:15.562938 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:15.564945 (+ 2007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:15.564948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:15.565034 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:36:15.565137 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:36:15.565261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":217,"spinlock_wait_cycles":2304,"thread_start_us":54,"threads_started":1}
I20260927 00:36:15.565699 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.551055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:35795 (local address 127.10.11.194:43633)
0927 00:36:15.551454 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:15.551488 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:15.551529 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:36:15.552168 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:15.552172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:15.552497 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:15.552793 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:15.554016 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.555056 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556245 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:15.556721 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:15.556890 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:15.560119 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:15.560125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:15.560139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:15.560172 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:15.562030 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:15.562585 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:15.562589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:15.562593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:15.562696 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:15.563039 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:15.563043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:15.563045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:15.563403 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:15.564908 (+ 1505us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:15.565287 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 00:36:15.565541 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":52,"threads_started":1}
W20260927 00:36:15.567366 25187 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
W20260927 00:36:15.567651 25189 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:15.567750 25189 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 601c11ab81954f2095240a3615e054b6; no voters: e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806
I20260927 00:36:15.567996 25572 raft_consensus.cc:2726] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:36:15.570920 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:15.570519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:47183)
0927 00:36:15.570752 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:15.570804 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260927 00:36:16.317883 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:16.317506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:42179)
0927 00:36:16.317739 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:16.317791 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 00:36:16.401417 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:16.401075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:33447)
0927 00:36:16.401295 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:16.401343 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260927 00:36:16.571909 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:16.571529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:60009)
0927 00:36:16.571785 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:16.571834 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260927 00:36:17.220798 25580 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:17.220904 25580 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:17.221349 25580 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
W20260927 00:36:17.222128 25189 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
W20260927 00:36:17.222244 25187 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:17.222348 25187 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 601c11ab81954f2095240a3615e054b6; no voters: e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806
I20260927 00:36:17.222491 25580 raft_consensus.cc:2726] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:36:17.318977 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:17.318597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:46815)
0927 00:36:17.318834 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:17.318886 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":67,"threads_started":1}
I20260927 00:36:17.402508 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:17.402088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:46147)
0927 00:36:17.402377 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:17.402429 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260927 00:36:17.573055 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:17.572615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:38579)
0927 00:36:17.572897 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:17.572951 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20260927 00:36:18.320091 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:18.319692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:33927)
0927 00:36:18.319944 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:18.319999 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260927 00:36:18.403527 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:18.403146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:55787)
0927 00:36:18.403393 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:18.403445 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260927 00:36:18.574180 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:18.573776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:53777)
0927 00:36:18.573997 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:18.574046 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260927 00:36:19.321360 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:19.320912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:45985)
0927 00:36:19.321216 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:19.321268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260927 00:36:19.404572 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:19.404232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:58403)
0927 00:36:19.404443 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:19.404491 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260927 00:36:19.575220 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:19.574863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:59081)
0927 00:36:19.575087 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:19.575136 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260927 00:36:20.247552 25590 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:20.247685 25590 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:20.248157 25590 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
W20260927 00:36:20.249044 25187 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
W20260927 00:36:20.249135 25189 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:20.249243 25189 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 601c11ab81954f2095240a3615e054b6; no voters: e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806
I20260927 00:36:20.249442 25590 raft_consensus.cc:2726] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:36:20.322603 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:20.322169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:36609)
0927 00:36:20.322437 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:20.322508 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260927 00:36:20.405697 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:20.405314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:41415)
0927 00:36:20.405549 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:20.405612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260927 00:36:20.576279 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:20.575899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:59625)
0927 00:36:20.576150 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:20.576201 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260927 00:36:21.323668 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:21.323277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:39721)
0927 00:36:21.323526 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:21.323579 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260927 00:36:21.406947 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:21.406519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:60219)
0927 00:36:21.406820 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:21.406869 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260927 00:36:21.577373 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:21.576973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:56915)
0927 00:36:21.577234 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:21.577288 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
I20260927 00:36:22.324766 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:22.324382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:33049)
0927 00:36:22.324620 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:22.324674 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260927 00:36:22.408020 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:22.407627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:45889)
0927 00:36:22.407886 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:22.407936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260927 00:36:22.578449 25599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:22.578024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:36809)
0927 00:36:22.578314 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:22.578363 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260927 00:36:23.325987 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:23.325527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:48795)
0927 00:36:23.325802 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:23.325857 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":115,"threads_started":1}
I20260927 00:36:23.409076 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:23.408714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:47917)
0927 00:36:23.408945 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:23.408995 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260927 00:36:23.579627 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:23.579195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:35507)
0927 00:36:23.579471 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:23.579530 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260927 00:36:24.327214 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:24.326836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:33357)
0927 00:36:24.327068 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:24.327121 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260927 00:36:24.361858 25604 raft_consensus.cc:482] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:24.362012 25604 raft_consensus.cc:504] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37493f0c3c341b2a96d17ee5c8d05ed" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 43633 } } peers { permanent_uuid: "f831a647546144b6be7f2b23ada08806" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33815 } } peers { permanent_uuid: "601c11ab81954f2095240a3615e054b6" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 42659 } }
I20260927 00:36:24.362694 25604 leader_election.cc:288] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633), f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815)
W20260927 00:36:24.363288 25189 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e37493f0c3c341b2a96d17ee5c8d05ed (127.10.11.194:43633): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
W20260927 00:36:24.363407 25187 leader_election.cc:341] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f831a647546144b6be7f2b23ada08806 (127.10.11.195:33815): Not found: Tablet not found: 0f2cf6a988024dfb9aa0fa31600eccdd
I20260927 00:36:24.363466 25187 leader_election.cc:302] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 601c11ab81954f2095240a3615e054b6; no voters: e37493f0c3c341b2a96d17ee5c8d05ed, f831a647546144b6be7f2b23ada08806
I20260927 00:36:24.363605 25604 raft_consensus.cc:2726] T 0f2cf6a988024dfb9aa0fa31600eccdd P 601c11ab81954f2095240a3615e054b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:36:24.410220 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:24.409794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:56901)
0927 00:36:24.410020 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:24.410070 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260927 00:36:24.580760 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:24.580362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:46185)
0927 00:36:24.580613 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:24.580671 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260927 00:36:25.328302 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:25.327913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:33565)
0927 00:36:25.328158 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:25.328212 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260927 00:36:25.411232 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:25.410863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:46521)
0927 00:36:25.411100 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:25.411149 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260927 00:36:25.581995 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:25.581601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:53653)
0927 00:36:25.581848 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:25.581901 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260927 00:36:26.329406 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:26.329007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:60587)
0927 00:36:26.329232 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:26.329284 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260927 00:36:26.412369 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:26.411959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:39011)
0927 00:36:26.412199 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:26.412257 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 00:36:26.583098 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:26.582715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:37775)
0927 00:36:26.582960 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:26.583012 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260927 00:36:27.330677 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:27.330223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:60723)
0927 00:36:27.330500 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:27.330564 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260927 00:36:27.413527 25614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:27.413140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:51115)
0927 00:36:27.413386 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:27.413436 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260927 00:36:27.584254 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:27.583833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:32995)
0927 00:36:27.584094 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:27.584159 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260927 00:36:28.331871 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:28.331473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:57151)
0927 00:36:28.331722 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:28.331777 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260927 00:36:28.414780 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:28.414365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:40739)
0927 00:36:28.414621 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:28.414677 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260927 00:36:28.585505 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:28.584985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.195:47969)
0927 00:36:28.585355 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:28.585415 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":88,"threads_started":1}
I20260927 00:36:29.333372 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:29.332927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.193:57693)
0927 00:36:29.333192 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:29.333253 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260927 00:36:29.416093 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:29.415674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:38669 (local address 127.10.11.194:34697)
0927 00:36:29.415944 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:29.416007 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260927 00:36:29.568008 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25136
I20260927 00:36:29.573971 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25302
I20260927 00:36:29.579234 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25437
2026-09-27T00:36:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19242 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:36:29 dist-test-slave-6gxc krb5kdc[25623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:36:29 dist-test-slave-6gxc krb5kdc[25623](info): set up 2 sockets
Sep 27 00:36:29 dist-test-slave-6gxc krb5kdc[25623](info): commencing operation
krb5kdc: starting...
W20260927 00:36:30.622331 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:36:30 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:36:30.745720 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:30.853349 25636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:30.853581 25636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:30.853619 25636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:30.857486 25636 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:36:30.857529 25636 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:36:30.857544 25636 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:36:30.857558 25636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:30.857575 25636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:30.857637 25636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:30.857667 25636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:30.857697 25636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:30.862798 25636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:30.863760 25636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:30.864732 25636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:36:30.872936 25636 server_base.cc:1034] running on GCE node
W20260927 00:36:30.872862 25642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:30.872881 25641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:30.872840 25644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:30.873803 25636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:30.874938 25636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:30.876116 25636 hybrid_clock.cc:648] HybridClock initialized: now 1790469390876072 us; error 57 us; skew 500 ppm
Sep 27 00:36:30 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469390, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:30.879012 25636 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:30.879925 25636 webserver.cc:460] Webserver started at http://127.10.11.254:37649/ using document root <none> and password file <none>
I20260927 00:36:30.880503 25636 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:30.880606 25636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:30.880789 25636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:30.882771 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "2c632205c25f44de8844b29edd69d699"
format_stamp: "Formatted at 2026-09-27 00:36:30 on dist-test-slave-6gxc"
server_key: "6f987aa721069972a872364ee91b54f5"
server_key_iv: "d06ffffa82d18984180c037cf2c2a305"
server_key_version: "encryptionkey@0"
I20260927 00:36:30.883267 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "2c632205c25f44de8844b29edd69d699"
format_stamp: "Formatted at 2026-09-27 00:36:30 on dist-test-slave-6gxc"
server_key: "6f987aa721069972a872364ee91b54f5"
server_key_iv: "d06ffffa82d18984180c037cf2c2a305"
server_key_version: "encryptionkey@0"
I20260927 00:36:30.886830 25636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:36:30.889111 25651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:30.890422 25636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:30.890563 25636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2c632205c25f44de8844b29edd69d699"
format_stamp: "Formatted at 2026-09-27 00:36:30 on dist-test-slave-6gxc"
server_key: "6f987aa721069972a872364ee91b54f5"
server_key_iv: "d06ffffa82d18984180c037cf2c2a305"
server_key_version: "encryptionkey@0"
I20260927 00:36:30.890677 25636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:30.899235 25636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:30.902019 25636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:30.902240 25636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:30.908887 25636 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:33691
I20260927 00:36:30.909160 25703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:33691 every 8 connection(s)
I20260927 00:36:30.909777 25636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:30.911252 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25636
I20260927 00:36:30.911391 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:36:30.911715 10287 external_mini_cluster.cc:1428] Setting key 45b2508d0b2cb35882581c64c3317edf
I20260927 00:36:30.912899 25704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:30.919128 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap starting.
Sep 27 00:36:30 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:30.921643 25704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:30.922577 25704 log.cc:824] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:30.924746 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: No bootstrap required, opened a new log
I20260927 00:36:30.925900 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:30.913004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.10.11.254:33691)
0927 00:36:30.913417 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:30.913438 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:30.913471 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:36:30.914088 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:30.914157 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:30.914472 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:30.914734 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:30.915597 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:30.916576 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:30.917755 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:30.918280 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:30.918403 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:30.920996 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:30.921002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:30.921017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:30.921048 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:30.922960 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:30.923461 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:30.923468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:30.923474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:30.923597 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:30.923856 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:30.923859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:30.923860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:30.924336 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:30.924517 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:30.924736 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:36:30.925012 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":144,"threads_started":1}
I20260927 00:36:30.928702 25704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:30.928900 25704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:30.928934 25704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Initialized, Role: FOLLOWER
I20260927 00:36:30.929270 25704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:30.929378 25704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:30.929414 25704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:30.929477 25704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:30.930270 25704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:30.930542 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c632205c25f44de8844b29edd69d699; no voters:
I20260927 00:36:30.930758 25704 leader_election.cc:288] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:36:30.930910 25709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:30.931188 25709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 LEADER]: Becoming Leader. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Running, Role: LEADER
I20260927 00:36:30.931493 25709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:30.931843 25704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:30.933163 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:36:30.933326 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:30.933708 25722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:30.933670 25711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c632205c25f44de8844b29edd69d699. Latest consensus state: current_term: 1 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:36:30.933753 25711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:30.936187 25722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:30.941305 25722 catalog_manager.cc:1347] Generated new cluster ID: f61ae7db5506448383057761c9bb05dd
I20260927 00:36:30.941368 25722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:30.954623 25722 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:36:30.955760 25722 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:30.965287 25722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Generated new TSK 0
I20260927 00:36:30.966009 25722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:36:30.996850 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:31.104004 25730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:31.104343 25730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:31.104422 25730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:31.108397 25730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:31.108476 25730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:31.108572 25730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:31.113586 25730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:31.114562 25730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:31.115491 25730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:31.124450 25735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:31.124464 25738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:31.124470 25736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:31.124516 25730 server_base.cc:1034] running on GCE node
I20260927 00:36:31.125043 25730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:31.125614 25730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:31.126767 25730 hybrid_clock.cc:648] HybridClock initialized: now 1790469391126754 us; error 24 us; skew 500 ppm
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:31.129293 25730 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:31.130093 25730 webserver.cc:460] Webserver started at http://127.10.11.193:39379/ using document root <none> and password file <none>
I20260927 00:36:31.130654 25730 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:31.130713 25730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:31.130932 25730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:31.132951 25730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "bc91bb392ef74817b4b5d56d8769bfad"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "e8aae053d6ad89ecbddbbee99095d178"
server_key_iv: "4100296a1d795f229df5cd0e9b21bcd9"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.133481 25730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "bc91bb392ef74817b4b5d56d8769bfad"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "e8aae053d6ad89ecbddbbee99095d178"
server_key_iv: "4100296a1d795f229df5cd0e9b21bcd9"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.136878 25730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:36:31.139065 25745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.140182 25730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:31.140308 25730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "bc91bb392ef74817b4b5d56d8769bfad"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "e8aae053d6ad89ecbddbbee99095d178"
server_key_iv: "4100296a1d795f229df5cd0e9b21bcd9"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.140407 25730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:31.163482 25730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:31.166103 25730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:31.166337 25730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:31.166867 25730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:31.167609 25730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:31.167654 25730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.167721 25730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:31.167752 25730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.178005 25730 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:45797
I20260927 00:36:31.178066 25858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:45797 every 8 connection(s)
I20260927 00:36:31.179039 25730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:31.182654 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25730
I20260927 00:36:31.182751 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:36:31.183043 10287 external_mini_cluster.cc:1428] Setting key c280ca79fc87a3c697f194c3babffb52
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:36:31.193655 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.180895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:43073 (local address 127.10.11.254:33691)
0927 00:36:31.181364 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.181369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.182319 (+ 950us) server_negotiation.cc:407] Connection header received
0927 00:36:31.183342 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.183346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.183401 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.183483 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:31.184540 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.185082 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.186661 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.186895 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.186981 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.189994 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:31.189996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:31.189998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:31.190020 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:31.191569 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.192191 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.192193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.192195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.192264 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.192696 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.192698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.192700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.192869 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:31.192946 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.193384 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.193540 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260927 00:36:31.194658 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.181279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:43073)
0927 00:36:31.182173 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.182224 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.183129 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.183637 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.183644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.183969 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:31.184398 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.184410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.185206 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.185209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.186529 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.186542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.187027 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.187034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.187265 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.187844 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:31.187868 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:31.189812 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:31.191693 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.191701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.191718 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.192092 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.192361 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.192364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.192366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.192547 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.192972 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:31.192976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:31.193184 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.193357 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.193710 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":510,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:36:31.195849 25859 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:31.196071 25859 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:31.196564 25859 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:31.198132 25668 ts_manager.cc:194] Registered new tserver with Master: bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797)
I20260927 00:36:31.199776 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:43073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:36:31.209036 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:31.314128 25864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:31.314396 25864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:31.314471 25864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:31.318419 25864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:31.318491 25864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:31.318583 25864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:31.323617 25864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:31.324574 25864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:31.325531 25864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:31.334697 25869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:31.334864 25864 server_base.cc:1034] running on GCE node
W20260927 00:36:31.334693 25870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:31.334693 25872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:31.335366 25864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:31.335949 25864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:31.337112 25864 hybrid_clock.cc:648] HybridClock initialized: now 1790469391337093 us; error 30 us; skew 500 ppm
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:31.339672 25864 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:31.340462 25864 webserver.cc:460] Webserver started at http://127.10.11.194:43305/ using document root <none> and password file <none>
I20260927 00:36:31.340992 25864 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:31.341080 25864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:31.341297 25864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:31.343266 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "f1b492e2690249128d5f4a1481346313"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "517557fec4e3f65442c77ad7f201a8d7"
server_key_iv: "4e6a660239a5192066b4cd988b3f4d67"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.343832 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "f1b492e2690249128d5f4a1481346313"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "517557fec4e3f65442c77ad7f201a8d7"
server_key_iv: "4e6a660239a5192066b4cd988b3f4d67"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.347332 25864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:36:31.349531 25879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.350703 25864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:31.350819 25864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "f1b492e2690249128d5f4a1481346313"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "517557fec4e3f65442c77ad7f201a8d7"
server_key_iv: "4e6a660239a5192066b4cd988b3f4d67"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.350914 25864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:31.367424 25864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:31.370484 25864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:31.370671 25864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:31.371205 25864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:31.371956 25864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:31.372000 25864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.372063 25864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:31.372102 25864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.381074 25864 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:36515
I20260927 00:36:31.381110 25992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:36515 every 8 connection(s)
I20260927 00:36:31.381898 25864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:31.384140 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25864
I20260927 00:36:31.384284 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:36:31.384559 10287 external_mini_cluster.cc:1428] Setting key 7b5f7dd4eec9dc7e68ed50fdd82b82fd
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:36:31.396605 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.383640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:46007 (local address 127.10.11.254:33691)
0927 00:36:31.383784 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.383789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.384847 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 00:36:31.385844 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.385847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.385899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.385989 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:31.387504 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.388357 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.390073 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.390362 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.390443 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.393080 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:31.393084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:31.393086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:31.393111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:31.394742 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.395226 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.395231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.395233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.395316 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.395696 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.395699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.395700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.395868 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:31.395943 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.396387 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.396482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:36:31.397329 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.384033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:46007)
0927 00:36:31.384708 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.384757 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.385614 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.386199 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.386207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.386700 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:31.387350 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.387370 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.388481 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.388484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.389945 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.389960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.390517 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.390521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.390730 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.391289 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:31.391314 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:31.392922 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:31.394854 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.394859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.394872 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.395119 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.395477 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.395480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.395481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.395605 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.395953 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:31.395957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:31.396149 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.396295 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.396509 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":137,"threads_started":1}
I20260927 00:36:31.398411 25993 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:31.398622 25993 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:31.399050 25993 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:31.400097 25668 ts_manager.cc:194] Registered new tserver with Master: f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
I20260927 00:36:31.400722 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:46007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:36:31.413479 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:31.521531 25998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:31.521795 25998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:31.521884 25998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:31.525823 25998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:31.525918 25998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:31.526021 25998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:31.531337 25998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:31.532331 25998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:31.533401 25998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:31.542757 26003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:31.542732 26004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:31.542814 25998 server_base.cc:1034] running on GCE node
W20260927 00:36:31.542732 26006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:31.543452 25998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:31.544054 25998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:31.545217 25998 hybrid_clock.cc:648] HybridClock initialized: now 1790469391545176 us; error 51 us; skew 500 ppm
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:31.547904 25998 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:31.548828 25998 webserver.cc:460] Webserver started at http://127.10.11.195:36095/ using document root <none> and password file <none>
I20260927 00:36:31.549369 25998 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:31.549461 25998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:31.549683 25998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:36:31.551643 25998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "bba18d08ad1d4f0425ff1576cc6b81f3"
server_key_iv: "736aedffbdb5d39d0d6395e3b05bbcac"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.552263 25998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "bba18d08ad1d4f0425ff1576cc6b81f3"
server_key_iv: "736aedffbdb5d39d0d6395e3b05bbcac"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.555887 25998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:36:31.558054 26013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.559376 25998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:31.559509 25998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "bba18d08ad1d4f0425ff1576cc6b81f3"
server_key_iv: "736aedffbdb5d39d0d6395e3b05bbcac"
server_key_version: "encryptionkey@0"
I20260927 00:36:31.559616 25998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:31.571404 25998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:31.573987 25998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:31.574196 25998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:31.574735 25998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:31.575488 25998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:31.575564 25998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.575644 25998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:31.575688 25998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:31.584036 25998 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:38845
I20260927 00:36:31.584107 26126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:38845 every 8 connection(s)
I20260927 00:36:31.584873 25998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:31.588897 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 25998
I20260927 00:36:31.589030 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:36:31.589272 10287 external_mini_cluster.cc:1428] Setting key 918ba7228737652e0fd53f5ce641abd9
Sep 27 00:36:31 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:31.597486 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.586536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:43591 (local address 127.10.11.254:33691)
0927 00:36:31.586703 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.586706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.587607 (+ 901us) server_negotiation.cc:407] Connection header received
0927 00:36:31.588429 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.588432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.588477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.588549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:31.589780 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.590455 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.591704 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.591915 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.591994 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.594263 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:31.594266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:31.594268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:31.594290 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:31.595581 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.596090 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.596093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.596095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.596191 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.596504 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.596507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.596508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.596694 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:31.596779 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.597254 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.597361 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 00:36:31.598274 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.586863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:43591)
0927 00:36:31.587468 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.587504 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.588234 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.588702 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.588708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.589096 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:31.589614 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.589627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.590576 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.590580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.591598 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.591607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.592032 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.592035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.592166 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.592651 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:31.592666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:31.594075 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:31.595702 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.595706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.595717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.595974 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.596286 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:31.596288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:31.596289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:31.596418 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:31.596810 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:31.596815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:31.597016 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.597189 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.597439 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":103,"threads_started":1}
I20260927 00:36:31.599267 26127 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:31.599467 26127 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:31.599846 26127 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:31.600807 25668 ts_manager.cc:194] Registered new tserver with Master: 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845)
I20260927 00:36:31.601419 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:43591
I20260927 00:36:31.603060 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:31.611810 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.605029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.10.11.254:33691)
0927 00:36:31.605154 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.605158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.605307 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:36:31.605404 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.605406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.605441 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.605501 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:31.606065 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.606611 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.607472 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.607669 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.607740 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.609143 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:31.609146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:31.609147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:31.609166 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:31.610417 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.610802 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.610804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.610805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.610873 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:31.611121 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:31.611123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:31.611124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:31.611302 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:31.611415 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.611624 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.611708 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:36:31.614705 25668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:36:31.617654 25668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:36:31.631508 26138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.625709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:38845 (local address 127.0.0.1:34122)
0927 00:36:31.626059 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.626130 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.626266 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.626962 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.626967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.626990 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:31.627125 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.627135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.628531 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.628534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.629826 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.629848 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.630937 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.630940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.631047 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631087 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.631206 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.631321 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
I20260927 00:36:31.631677 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.626471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:36515 (local address 127.0.0.1:54452)
0927 00:36:31.627031 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.627047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.627139 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.627737 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.627740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.627757 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:31.627870 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.627879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.629075 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.629079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.630300 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.630311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.631255 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.631259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.631356 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631382 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.631470 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.631540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":22,"thread_start_us":83,"threads_started":1}
I20260927 00:36:31.632078 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.625896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.10.11.195:38845)
0927 00:36:31.626235 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.626247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.626270 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:36:31.626405 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.626411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.626610 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.626780 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:31.627264 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.628386 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.629964 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.630808 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.630915 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.631833 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.631945 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260927 00:36:31.632093 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.626765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45797 (local address 127.0.0.1:39362)
0927 00:36:31.627208 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.627222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.627293 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.627946 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.627951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.627972 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:31.628060 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.628069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.629433 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.629437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.630481 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.630492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.631680 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.631685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.631832 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631857 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.631923 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.631981 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260927 00:36:31.632095 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.626623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.10.11.194:36515)
0927 00:36:31.627392 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.627409 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.627428 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:36:31.627487 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.627490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.627612 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.627758 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:31.628028 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.628936 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.630429 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.631126 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.631227 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631344 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.631512 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.631992 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":124,"threads_started":1}
I20260927 00:36:31.632535 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.626975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.10.11.193:45797)
0927 00:36:31.627408 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.627418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.627442 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:36:31.627515 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.627520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.627700 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.627843 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:31.628188 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.629278 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.630610 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.631412 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.631505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.631621 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.632251 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.632348 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":135,"threads_started":1}
I20260927 00:36:31.635006 25793 tablet_service.cc:1467] Processing CreateTablet for tablet e585b8ae8500464cbf6a3364971cec8e (DEFAULT_TABLE table=test-table [id=195104f7aed84a74b2475fb13961f999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:31.635053 26061 tablet_service.cc:1467] Processing CreateTablet for tablet e585b8ae8500464cbf6a3364971cec8e (DEFAULT_TABLE table=test-table [id=195104f7aed84a74b2475fb13961f999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:31.635514 25927 tablet_service.cc:1467] Processing CreateTablet for tablet e585b8ae8500464cbf6a3364971cec8e (DEFAULT_TABLE table=test-table [id=195104f7aed84a74b2475fb13961f999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:31.636063 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e585b8ae8500464cbf6a3364971cec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:31.636078 26061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e585b8ae8500464cbf6a3364971cec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:31.636363 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e585b8ae8500464cbf6a3364971cec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:31.641945 26144 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap starting.
I20260927 00:36:31.643359 26145 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Bootstrap starting.
I20260927 00:36:31.644208 26144 tablet_bootstrap.cc:654] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:31.645304 26144 log.cc:824] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:31.645402 26145 tablet_bootstrap.cc:654] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:31.646139 26145 log.cc:824] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:31.646791 26146 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Bootstrap starting.
I20260927 00:36:31.647517 26144 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: No bootstrap required, opened a new log
I20260927 00:36:31.647720 26144 ts_tablet_manager.cc:1397] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:36:31.648878 26146 tablet_bootstrap.cc:654] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:31.649520 26146 log.cc:824] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:31.650929 26144 raft_consensus.cc:348] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.651079 26145 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: No bootstrap required, opened a new log
I20260927 00:36:31.651158 26144 raft_consensus.cc:374] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:31.651230 26144 raft_consensus.cc:729] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc91bb392ef74817b4b5d56d8769bfad, State: Initialized, Role: FOLLOWER
I20260927 00:36:31.651263 26145 ts_tablet_manager.cc:1397] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 00:36:31.651301 26146 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: No bootstrap required, opened a new log
I20260927 00:36:31.651477 26146 ts_tablet_manager.cc:1397] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:36:31.651688 26144 consensus_queue.cc:260] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.652537 26144 ts_tablet_manager.cc:1428] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:36:31.652892 25859 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
I20260927 00:36:31.655313 26145 raft_consensus.cc:348] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.655313 26146 raft_consensus.cc:348] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.655578 26146 raft_consensus.cc:374] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:31.655578 26145 raft_consensus.cc:374] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:31.655637 26146 raft_consensus.cc:729] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b492e2690249128d5f4a1481346313, State: Initialized, Role: FOLLOWER
I20260927 00:36:31.655637 26145 raft_consensus.cc:729] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115eae9ccfc424894efd34a9bc3d6ac, State: Initialized, Role: FOLLOWER
I20260927 00:36:31.656087 26145 consensus_queue.cc:260] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.656087 26146 consensus_queue.cc:260] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.656953 25993 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
I20260927 00:36:31.656997 26127 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
I20260927 00:36:31.657023 26146 ts_tablet_manager.cc:1428] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:36:31.657023 26145 ts_tablet_manager.cc:1428] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260927 00:36:31.680536 25860 tablet.cc:2369] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:31.804189 26150 raft_consensus.cc:482] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:31.804418 26150 raft_consensus.cc:504] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.805634 26150 leader_election.cc:288] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
I20260927 00:36:31.810407 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.805994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:59565 (local address 127.10.11.195:38845)
0927 00:36:31.806168 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.806174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.806308 (+ 134us) server_negotiation.cc:407] Connection header received
0927 00:36:31.806451 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.806455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.806502 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.806581 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:31.806886 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.807686 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.809275 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.809801 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.809931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.809982 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.810189 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.810268 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:36:31.810431 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.805844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:38845 (local address 127.10.11.193:59565)
0927 00:36:31.806167 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.806184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.806338 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.806600 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.806603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.806624 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:31.806768 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.806778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.807789 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.807792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.809175 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.809182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.809923 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.809926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.810030 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.810054 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.810168 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.810244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":45,"thread_start_us":90,"threads_started":1}
I20260927 00:36:31.810672 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.806457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:46625 (local address 127.10.11.194:36515)
0927 00:36:31.806604 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.806608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.806697 (+ 89us) server_negotiation.cc:407] Connection header received
0927 00:36:31.806807 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.806809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.806887 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.806965 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:31.807190 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.807881 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.809326 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.809830 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.809940 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.809989 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.810351 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.810425 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:36:31.810734 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.806311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:36515 (local address 127.10.11.193:46625)
0927 00:36:31.806626 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.806639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:31.806729 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:31.806986 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:31.806988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:31.807003 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:31.807088 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.807092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.807990 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.807993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.809163 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:31.809175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.810053 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.810057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:31.810207 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.810232 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:36:31.810311 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:36:31.810379 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260927 00:36:31.811041 26081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e585b8ae8500464cbf6a3364971cec8e" candidate_uuid: "bc91bb392ef74817b4b5d56d8769bfad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" is_pre_election: true
I20260927 00:36:31.811311 25947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e585b8ae8500464cbf6a3364971cec8e" candidate_uuid: "bc91bb392ef74817b4b5d56d8769bfad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b492e2690249128d5f4a1481346313" is_pre_election: true
I20260927 00:36:31.811394 26081 raft_consensus.cc:2454] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc91bb392ef74817b4b5d56d8769bfad in term 0.
I20260927 00:36:31.811587 25947 raft_consensus.cc:2454] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc91bb392ef74817b4b5d56d8769bfad in term 0.
I20260927 00:36:31.811880 25748 leader_election.cc:302] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115eae9ccfc424894efd34a9bc3d6ac, bc91bb392ef74817b4b5d56d8769bfad; no voters:
I20260927 00:36:31.812142 26150 raft_consensus.cc:2781] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:31.812215 26150 raft_consensus.cc:482] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:31.812245 26150 raft_consensus.cc:3037] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:31.813279 26150 raft_consensus.cc:504] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.813603 26150 leader_election.cc:288] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 1 election: Requested vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
I20260927 00:36:31.813939 26081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e585b8ae8500464cbf6a3364971cec8e" candidate_uuid: "bc91bb392ef74817b4b5d56d8769bfad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
I20260927 00:36:31.814072 25947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e585b8ae8500464cbf6a3364971cec8e" candidate_uuid: "bc91bb392ef74817b4b5d56d8769bfad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b492e2690249128d5f4a1481346313"
I20260927 00:36:31.814079 26081 raft_consensus.cc:3037] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:31.814237 25947 raft_consensus.cc:3037] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:31.815116 26081 raft_consensus.cc:2454] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc91bb392ef74817b4b5d56d8769bfad in term 1.
I20260927 00:36:31.815248 25947 raft_consensus.cc:2454] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc91bb392ef74817b4b5d56d8769bfad in term 1.
I20260927 00:36:31.815475 25748 leader_election.cc:302] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115eae9ccfc424894efd34a9bc3d6ac, bc91bb392ef74817b4b5d56d8769bfad; no voters:
I20260927 00:36:31.815717 26150 raft_consensus.cc:2781] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:31.815924 26150 raft_consensus.cc:686] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 LEADER]: Becoming Leader. State: Replica: bc91bb392ef74817b4b5d56d8769bfad, State: Running, Role: LEADER
I20260927 00:36:31.816256 26150 consensus_queue.cc:237] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:31.819311 25668 catalog_manager.cc:5495] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad reported cstate change: term changed from 0 to 1, leader changed from <none> to bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193). New cstate: current_term: 1 leader_uuid: "bc91bb392ef74817b4b5d56d8769bfad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } health_report { overall_health: HEALTHY } } }
W20260927 00:36:31.835737 26128 tablet.cc:2369] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:31.863754 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.860035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.10.11.193:45797)
0927 00:36:31.860177 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:31.860181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:31.860197 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:36:31.860341 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:31.860348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:31.860404 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:31.860523 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:31.860768 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.861264 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.862214 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:31.862496 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:31.862613 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:31.862939 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:31.863345 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:31.863480 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:31.863555 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 00:36:31.863611 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:36:31.869438 25947 raft_consensus.cc:1261] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Refusing update from remote peer bc91bb392ef74817b4b5d56d8769bfad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:31.869455 26081 raft_consensus.cc:1261] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Refusing update from remote peer bc91bb392ef74817b4b5d56d8769bfad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:31.870167 26155 consensus_queue.cc:1035] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:31.870398 26150 consensus_queue.cc:1035] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:31.877614 26160 mvcc.cc:205] Tried to move back new op lower bound from 7333762629090082816 to 7333762628881838080. Current Snapshot: MvccSnapshot[applied={T|T < 7333762629090082816}]
I20260927 00:36:31.884680 25668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:31.884829 25668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:31.887006 25668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Sending DeleteTablet for 3 replicas of tablet e585b8ae8500464cbf6a3364971cec8e
I20260927 00:36:31.888026 25927 tablet_service.cc:1514] Processing DeleteTablet for tablet e585b8ae8500464cbf6a3364971cec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:54452
I20260927 00:36:31.888067 26061 tablet_service.cc:1514] Processing DeleteTablet for tablet e585b8ae8500464cbf6a3364971cec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:34122
I20260927 00:36:31.888187 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25730
I20260927 00:36:31.889030 26168 tablet_replica.cc:330] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: stopping tablet replica
I20260927 00:36:31.889344 26168 raft_consensus.cc:2229] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:31.889600 26168 raft_consensus.cc:2258] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:31.889931 26167 tablet_replica.cc:330] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: stopping tablet replica
I20260927 00:36:31.890244 26167 raft_consensus.cc:2229] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:31.890516 26167 raft_consensus.cc:2258] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:31.891007 26168 ts_tablet_manager.cc:1905] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:31.891848 26167 ts_tablet_manager.cc:1905] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:31.893694 26168 ts_tablet_manager.cc:1918] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:31.893800 26168 log.cc:1196] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/e585b8ae8500464cbf6a3364971cec8e
I20260927 00:36:31.894134 26168 ts_tablet_manager.cc:1939] T e585b8ae8500464cbf6a3364971cec8e P f1b492e2690249128d5f4a1481346313: Deleting consensus metadata
I20260927 00:36:31.894618 26167 ts_tablet_manager.cc:1918] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:31.894702 26167 log.cc:1196] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/e585b8ae8500464cbf6a3364971cec8e
I20260927 00:36:31.894713 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25864
I20260927 00:36:31.895505 26167 ts_tablet_manager.cc:1939] T e585b8ae8500464cbf6a3364971cec8e P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting consensus metadata
I20260927 00:36:31.896556 25654 catalog_manager.cc:4841] TS 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): tablet e585b8ae8500464cbf6a3364971cec8e (table test-table [id=195104f7aed84a74b2475fb13961f999]) successfully deleted
W20260927 00:36:31.896761 25654 connection.cc:535] client connection to 127.10.11.193:45797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 00:36:31.896903 25654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:36:31.897838 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.897579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45797 (local address 127.0.0.1:39378)
0927 00:36:31.897705 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.897767 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:36:31.897974 25654 catalog_manager.cc:4584] TS bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e585b8ae8500464cbf6a3364971cec8e: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45797: connect: Connection refused (error 111)
I20260927 00:36:31.900777 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25998
I20260927 00:36:31.902662 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:31.900967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:36515 (local address 127.0.0.1:54464)
0927 00:36:31.902556 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:31.902602 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.194:36515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1519}
I20260927 00:36:31.906447 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 25636
I20260927 00:36:51.912688 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--webserver_interface=127.10.11.254
--webserver_port=37649
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:36:52.018764 26170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:52.019002 26170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:52.019040 26170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:52.022974 26170 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:36:52.023015 26170 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:36:52.023031 26170 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:36:52.023044 26170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:36:52.023062 26170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:52.023121 26170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:52.023149 26170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:36:52.023180 26170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:36:52.028333 26170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=37649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:52.029259 26170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:52.030233 26170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:52.042698 26176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.042689 26177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.042727 26179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:52.042708 26170 server_base.cc:1034] running on GCE node
I20260927 00:36:52.043574 26170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:52.044590 26170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:52.045769 26170 hybrid_clock.cc:648] HybridClock initialized: now 1790469412045742 us; error 39 us; skew 500 ppm
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:52.053666 26170 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:36:52.054548 26170 webserver.cc:460] Webserver started at http://127.10.11.254:37649/ using document root <none> and password file <none>
I20260927 00:36:52.055065 26170 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:52.055131 26170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:52.058804 26170 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:36:52.060827 26186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.061986 26170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:52.062114 26170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2c632205c25f44de8844b29edd69d699"
format_stamp: "Formatted at 2026-09-27 00:36:30 on dist-test-slave-6gxc"
server_key: "6f987aa721069972a872364ee91b54f5"
server_key_iv: "d06ffffa82d18984180c037cf2c2a305"
server_key_version: "encryptionkey@0"
I20260927 00:36:52.062700 26170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:52.085947 26170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:52.088898 26170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:52.089089 26170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:52.095990 26170 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:33691
I20260927 00:36:52.096278 26238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:33691 every 8 connection(s)
I20260927 00:36:52.097049 26170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:36:52.098739 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26170
I20260927 00:36:52.099162 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:45797
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=39379
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:52.101771 26239 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:36:52.103592 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap starting.
I20260927 00:36:52.120260 26239 log.cc:824] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:52.128612 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:52.128989 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap complete.
I20260927 00:36:52.132912 26239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:52.133456 26239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Initialized, Role: FOLLOWER
I20260927 00:36:52.134038 26239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:52.134202 26239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:36:52.134272 26239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:36:52.134364 26239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:36:52.136160 26239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:52.136490 26239 leader_election.cc:302] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c632205c25f44de8844b29edd69d699; no voters:
I20260927 00:36:52.136816 26239 leader_election.cc:288] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:36:52.136893 26243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:36:52.137116 26243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 LEADER]: Becoming Leader. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Running, Role: LEADER
I20260927 00:36:52.137393 26243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:36:52.138018 26239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:36:52.138345 26245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c632205c25f44de8844b29edd69d699. Latest consensus state: current_term: 2 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:36:52.138478 26245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:52.138700 26244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:36:52.138806 26244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:36:52.138818 26247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:36:52.143534 26247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:36:52.144130 26247 catalog_manager.cc:1259] Loaded cluster ID: f61ae7db5506448383057761c9bb05dd
I20260927 00:36:52.144194 26247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:36:52.146667 26247 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:36:52.147337 26247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Loaded TSK: 0
I20260927 00:36:52.161912 26247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Generated new TSK 1
I20260927 00:36:52.162184 26247 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:36:52.211308 26241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:52.211566 26241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:52.211638 26241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:52.215631 26241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:52.215703 26241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:52.215786 26241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:36:52.220825 26241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:45797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=39379
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:52.221776 26241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:52.222776 26241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:52.232942 26266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.233001 26269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.232955 26267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:52.233559 26241 server_base.cc:1034] running on GCE node
I20260927 00:36:52.233946 26241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:52.234584 26241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:52.235764 26241 hybrid_clock.cc:648] HybridClock initialized: now 1790469412235741 us; error 50 us; skew 500 ppm
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:52.238600 26241 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:36:52.239658 26241 webserver.cc:460] Webserver started at http://127.10.11.193:39379/ using document root <none> and password file <none>
I20260927 00:36:52.240343 26241 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:52.240442 26241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:52.245242 26241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:52.247750 26276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.249200 26241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:52.249318 26241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "bc91bb392ef74817b4b5d56d8769bfad"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "e8aae053d6ad89ecbddbbee99095d178"
server_key_iv: "4100296a1d795f229df5cd0e9b21bcd9"
server_key_version: "encryptionkey@0"
I20260927 00:36:52.249974 26241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:52.283078 26241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:52.286372 26241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:52.286569 26241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:52.287144 26241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:52.288293 26283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:36:52.292174 26241 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:36:52.292253 26241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 00:36:52.292361 26241 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:36:52.293948 26241 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:36:52.294013 26241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:36:52.294330 26283 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap starting.
I20260927 00:36:52.304064 26241 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:45797
I20260927 00:36:52.304087 26390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:45797 every 8 connection(s)
I20260927 00:36:52.304930 26241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:36:52.305893 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26241
I20260927 00:36:52.306419 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:36515
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=43305
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:36:52.313726 26283 log.cc:824] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:52.321166 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.307063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:38481)
0927 00:36:52.307869 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.307922 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.308726 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.309824 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.309832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.310308 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:52.310740 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.310753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.312114 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.312117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.312950 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.312959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.313684 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.313688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.313938 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.314723 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:52.314741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:52.316105 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:52.318209 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.318215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.318234 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.318480 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.318819 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.318825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.318830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.318972 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.319542 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:52.319547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:52.319782 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.319965 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.320289 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":145,"threads_started":1}
I20260927 00:36:52.321154 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.307127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:38481 (local address 127.10.11.254:33691)
0927 00:36:52.307590 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.307612 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.308067 (+ 455us) server_negotiation.cc:407] Connection header received
0927 00:36:52.309144 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.309155 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.309443 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.309757 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.310913 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.311966 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.313073 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.313543 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.313712 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.316295 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.316300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.316313 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.316339 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.318016 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.318599 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.318609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.318613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.318722 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.319076 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.319079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.319081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.319420 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.319557 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.320078 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.320324 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":119,"threads_started":1}
I20260927 00:36:52.322691 26283 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:36:52.323103 26283 tablet_bootstrap.cc:492] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap complete.
I20260927 00:36:52.323658 26391 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:52.323889 26391 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:52.324342 26391 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:52.324512 26283 ts_tablet_manager.cc:1397] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.003s
I20260927 00:36:52.326870 26203 ts_manager.cc:194] Registered new tserver with Master: bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797)
I20260927 00:36:52.327425 26283 raft_consensus.cc:348] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.327908 26283 raft_consensus.cc:729] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc91bb392ef74817b4b5d56d8769bfad, State: Initialized, Role: FOLLOWER
I20260927 00:36:52.328387 26283 consensus_queue.cc:260] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.329289 26283 ts_tablet_manager.cc:1428] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260927 00:36:52.330403 26203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:38481
I20260927 00:36:52.332711 26391 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.193@KRBTEST.COM
I20260927 00:36:52.340251 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.329712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45797 (local address 127.0.0.1:52246)
0927 00:36:52.330434 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.330476 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.330593 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.331260 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.331267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.331684 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:52.331815 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.331824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.333051 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.333055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.333760 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.333770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.334518 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.334522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.334637 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.335137 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:52.335149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:52.336437 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:52.338373 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.338383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.338388 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.338735 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.339058 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.339061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.339063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.339150 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.339616 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:52.339622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:52.339746 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.339884 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.340067 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":101,"threads_started":1}
I20260927 00:36:52.340261 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.329800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.10.11.193:45797)
0927 00:36:52.330139 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.330155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.330719 (+ 564us) server_negotiation.cc:407] Connection header received
0927 00:36:52.330784 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.330789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.330983 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.331237 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.331971 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.332952 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.333880 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.334395 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.334518 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.336565 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.336574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.336585 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.336608 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.338262 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.338853 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.338859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.338860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.338957 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.339239 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.339243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.339245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.339494 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.339639 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.339973 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.340123 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260927 00:36:52.341346 26325 tablet_service.cc:1514] Processing DeleteTablet for tablet e585b8ae8500464cbf6a3364971cec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:52246
I20260927 00:36:52.342301 26400 tablet_replica.cc:330] stopping tablet replica
I20260927 00:36:52.342589 26400 raft_consensus.cc:2229] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:52.342825 26400 raft_consensus.cc:2258] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:52.343722 26400 ts_tablet_manager.cc:1905] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:52.347016 26400 ts_tablet_manager.cc:1918] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:52.347124 26400 log.cc:1196] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal/wals/e585b8ae8500464cbf6a3364971cec8e
I20260927 00:36:52.347388 26400 ts_tablet_manager.cc:1939] T e585b8ae8500464cbf6a3364971cec8e P bc91bb392ef74817b4b5d56d8769bfad: Deleting consensus metadata
I20260927 00:36:52.348949 26189 catalog_manager.cc:4841] TS bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797): tablet e585b8ae8500464cbf6a3364971cec8e (table test-table [id=195104f7aed84a74b2475fb13961f999]) successfully deleted
W20260927 00:36:52.422145 26395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:52.422372 26395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:52.422411 26395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:52.426467 26395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:52.426545 26395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:52.426651 26395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:36:52.432102 26395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:36515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=43305
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:52.433022 26395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:52.433923 26395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:36:52.443128 26405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.443162 26406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.443158 26408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:52.443411 26395 server_base.cc:1034] running on GCE node
I20260927 00:36:52.443820 26395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:52.444375 26395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:52.445556 26395 hybrid_clock.cc:648] HybridClock initialized: now 1790469412445549 us; error 31 us; skew 500 ppm
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:52.448052 26395 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:36:52.448801 26395 webserver.cc:460] Webserver started at http://127.10.11.194:43305/ using document root <none> and password file <none>
I20260927 00:36:52.449318 26395 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:52.449419 26395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:52.452839 26395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:52.454732 26415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.455941 26395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:36:52.456059 26395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "f1b492e2690249128d5f4a1481346313"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "517557fec4e3f65442c77ad7f201a8d7"
server_key_iv: "4e6a660239a5192066b4cd988b3f4d67"
server_key_version: "encryptionkey@0"
I20260927 00:36:52.456573 26395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:52.473412 26395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:52.476325 26395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:52.476536 26395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:52.477072 26395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:52.477857 26395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:52.477932 26395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.478003 26395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:52.478039 26395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.486260 26395 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:36515
I20260927 00:36:52.486284 26528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:36515 every 8 connection(s)
I20260927 00:36:52.487143 26395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:36:52.494234 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26395
I20260927 00:36:52.494635 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:38845
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=36095
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:52.499326 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.488760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:55355 (local address 127.10.11.254:33691)
0927 00:36:52.488898 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.488903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.489896 (+ 993us) server_negotiation.cc:407] Connection header received
0927 00:36:52.490683 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.490685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.490734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.490812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.491828 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.492344 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.493354 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.493578 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.493658 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.495984 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.495988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.495991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.496021 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.497366 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.497838 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.497841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.497843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.497928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.498309 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.498314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.498316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.498489 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.498596 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.499036 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.499165 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:36:52.500033 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.489170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:55355)
0927 00:36:52.489768 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.489804 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.490516 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.490959 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.490965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.491273 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:52.491702 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.491713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.492443 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.492445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.493253 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.493261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.493696 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.493699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.493824 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.494430 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:52.494445 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:52.495825 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:52.497480 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.497485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.497498 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.497732 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.498017 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.498021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.498024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.498218 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.498592 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:52.498598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:52.498814 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.498979 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.499258 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":108,"threads_started":1}
I20260927 00:36:52.501121 26529 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:52.501346 26529 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:52.501765 26529 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:52.502808 26203 ts_manager.cc:194] Registered new tserver with Master: f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
I20260927 00:36:52.503691 26203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:55355
W20260927 00:36:52.605615 26532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:36:52.605849 26532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:36:52.605890 26532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:36:52.609938 26532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:36:52.609988 26532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:36:52.610044 26532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:36:52.615067 26532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:38845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=36095
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:36:52.615937 26532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:36:52.616829 26532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:36:52.625963 26532 server_base.cc:1034] running on GCE node
W20260927 00:36:52.625932 26538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.625929 26537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:36:52.625933 26540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:36:52.626544 26532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:36:52.627094 26532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:36:52.628262 26532 hybrid_clock.cc:648] HybridClock initialized: now 1790469412628232 us; error 41 us; skew 500 ppm
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:36:52.631011 26532 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:36:52.631951 26532 webserver.cc:460] Webserver started at http://127.10.11.195:36095/ using document root <none> and password file <none>
I20260927 00:36:52.632498 26532 fs_manager.cc:359] Metadata directory not provided
I20260927 00:36:52.632606 26532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:36:52.636292 26532 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:36:52.638377 26547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.639935 26532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:52.640057 26532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "bba18d08ad1d4f0425ff1576cc6b81f3"
server_key_iv: "736aedffbdb5d39d0d6395e3b05bbcac"
server_key_version: "encryptionkey@0"
I20260927 00:36:52.640646 26532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:36:52.674078 26532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:36:52.678203 26532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:36:52.678419 26532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:36:52.679003 26532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:36:52.679883 26532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:36:52.679948 26532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.680019 26532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:36:52.680052 26532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:36:52.692198 26532 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:38845
I20260927 00:36:52.692263 26660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:38845 every 8 connection(s)
I20260927 00:36:52.693392 26532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:36:52.701040 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26532
Sep 27 00:36:52 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469412, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:36:52.709880 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.696868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:45131 (local address 127.10.11.254:33691)
0927 00:36:52.696995 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.696999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.697022 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:36:52.698306 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.698310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.698370 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.698467 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.699596 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.700215 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.701677 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.701996 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.702057 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.705110 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.705114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.705117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.705143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.707287 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.707949 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.707953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.707955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.708064 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.708486 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.708489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.708491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.708742 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.708835 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.709487 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.709669 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:36:52.710988 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.695944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:45131)
0927 00:36:52.696777 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.696834 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.698014 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.698642 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.698649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.698993 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:36:52.699450 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.699463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.700336 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.700339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.701536 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.701551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.702268 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.702271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.702411 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.703100 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:36:52.703116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:36:52.704918 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:36:52.707441 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.707447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.707462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.707825 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.708165 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:36:52.708169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:36:52.708173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:36:52.708376 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:36:52.708863 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:36:52.708870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:36:52.709204 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.709443 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.709806 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":139,"threads_started":1}
I20260927 00:36:52.712090 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.701686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.10.11.254:33691)
0927 00:36:52.702002 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.702006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.702154 (+ 148us) server_negotiation.cc:407] Connection header received
0927 00:36:52.702342 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.702345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.702397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.702496 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.703388 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.704234 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.705273 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.705692 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.705803 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.707646 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.707652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.707657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.707682 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.709863 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.710617 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.710620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.710622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.710718 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.711006 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.711011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.711013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.711223 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.711429 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.711801 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.711909 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":69,"threads_started":1}
I20260927 00:36:52.712116 26661 heartbeater.cc:347] Connected to a master server at 127.10.11.254:33691
I20260927 00:36:52.712337 26661 heartbeater.cc:464] Registering TS with master...
I20260927 00:36:52.712898 26661 heartbeater.cc:511] Master 127.10.11.254:33691 requested a full tablet report, sending...
I20260927 00:36:52.714021 26202 ts_manager.cc:194] Registered new tserver with Master: 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845)
I20260927 00:36:52.714886 26202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:45131
I20260927 00:36:52.725970 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:36:52.730202 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.726610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.10.11.254:33691)
0927 00:36:52.726756 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.726762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.726999 (+ 237us) server_negotiation.cc:407] Connection header received
0927 00:36:52.727181 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.727183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.727330 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.727421 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:52.727800 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.728367 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.729202 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.729420 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.729497 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.729737 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:52.729858 (+ 121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:36:52.730041 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43472: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:36:52.730268 26664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43472: token expired
W20260927 00:36:52.730556 26131 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.11.254:33691)
I20260927 00:36:52.736857 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.730905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.10.11.254:33691)
0927 00:36:52.731012 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.731015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.731026 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:36:52.731078 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.731080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.731110 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.731172 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:36:52.731687 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.732135 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.732885 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.733084 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.733171 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.734389 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:36:52.734392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:36:52.734393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:36:52.734414 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:36:52.735670 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.736022 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.736024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.736025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.736092 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:36:52.736302 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:36:52.736304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:36:52.736305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:36:52.736449 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:36:52.736543 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.736689 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.736766 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:36:52.739104 26202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:36:52.742388 26202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:36:52.750698 26325 tablet_service.cc:1467] Processing CreateTablet for tablet c0562fe88b554763bca3af8f05e9973a (DEFAULT_TABLE table=test-table [id=b364fa1793c54a62978632d93ed61aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:52.751346 26325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0562fe88b554763bca3af8f05e9973a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:52.753922 26283 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap starting.
I20260927 00:36:52.754612 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.749449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:38845 (local address 127.0.0.1:60680)
0927 00:36:52.749935 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.749949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.750038 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.750707 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.750709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.750726 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:52.750829 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.750834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.751924 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.751927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.752941 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.752951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.754215 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.754219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.754308 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.754356 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.754426 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.754495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427}
I20260927 00:36:52.754884 26283 tablet_bootstrap.cc:654] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:52.755081 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.749702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:36515 (local address 127.0.0.1:55182)
0927 00:36:52.749970 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.749987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.750067 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.750880 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.750883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.750901 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:52.750985 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.750991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.752415 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.752419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.753509 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.753518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.754629 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.754633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.754766 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.754803 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.754855 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.754931 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260927 00:36:52.755611 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.749591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.10.11.195:38845)
0927 00:36:52.750318 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.750327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.750346 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:36:52.750404 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.750409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.750569 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.750732 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:52.750999 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.751810 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.753076 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.753911 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.755036 (+ 1125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.755316 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.755372 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.755470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":157,"threads_started":1}
I20260927 00:36:52.755640 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.749869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.10.11.194:36515)
0927 00:36:52.750389 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.750401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.750423 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:36:52.750499 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.750503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.750705 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.751129 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:52.751296 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.752289 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.753669 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.754501 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.755248 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.755399 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.755449 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.755533 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":125,"threads_started":1}
I20260927 00:36:52.756613 26283 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: No bootstrap required, opened a new log
I20260927 00:36:52.756705 26283 ts_tablet_manager.cc:1397] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:36:52.757130 26283 raft_consensus.cc:348] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.757249 26283 raft_consensus.cc:374] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:52.757287 26283 raft_consensus.cc:729] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc91bb392ef74817b4b5d56d8769bfad, State: Initialized, Role: FOLLOWER
I20260927 00:36:52.757445 26283 consensus_queue.cc:260] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.757777 26283 ts_tablet_manager.cc:1428] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:36:52.758561 26463 tablet_service.cc:1467] Processing CreateTablet for tablet c0562fe88b554763bca3af8f05e9973a (DEFAULT_TABLE table=test-table [id=b364fa1793c54a62978632d93ed61aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:52.758561 26589 tablet_service.cc:1467] Processing CreateTablet for tablet c0562fe88b554763bca3af8f05e9973a (DEFAULT_TABLE table=test-table [id=b364fa1793c54a62978632d93ed61aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:36:52.759585 26589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0562fe88b554763bca3af8f05e9973a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:52.759616 26463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0562fe88b554763bca3af8f05e9973a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:36:52.766266 26674 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Bootstrap starting.
I20260927 00:36:52.766450 26675 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Bootstrap starting.
I20260927 00:36:52.768555 26674 tablet_bootstrap.cc:654] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:52.768560 26675 tablet_bootstrap.cc:654] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Neither blocks nor log segments found. Creating new log.
I20260927 00:36:52.769253 26675 log.cc:824] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:52.769253 26674 log.cc:824] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Log is configured to *not* fsync() on all Append() calls
I20260927 00:36:52.771325 26674 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: No bootstrap required, opened a new log
I20260927 00:36:52.771516 26674 ts_tablet_manager.cc:1397] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:36:52.771557 26675 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: No bootstrap required, opened a new log
I20260927 00:36:52.771745 26675 ts_tablet_manager.cc:1397] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:36:52.774328 26674 raft_consensus.cc:348] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.774549 26674 raft_consensus.cc:374] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:52.774398 26675 raft_consensus.cc:348] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.774606 26674 raft_consensus.cc:729] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115eae9ccfc424894efd34a9bc3d6ac, State: Initialized, Role: FOLLOWER
I20260927 00:36:52.774639 26675 raft_consensus.cc:374] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:36:52.774699 26675 raft_consensus.cc:729] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b492e2690249128d5f4a1481346313, State: Initialized, Role: FOLLOWER
I20260927 00:36:52.775028 26674 consensus_queue.cc:260] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.775101 26675 consensus_queue.cc:260] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.775621 26661 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
I20260927 00:36:52.775987 26674 ts_tablet_manager.cc:1428] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260927 00:36:52.776011 26675 ts_tablet_manager.cc:1428] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 00:36:52.776353 26529 heartbeater.cc:503] Master 127.10.11.254:33691 was elected leader, sending a full tablet report...
W20260927 00:36:52.806927 26392 tablet.cc:2369] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:36:52.813488 26679 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:36:52.813705 26679 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.814874 26679 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797)
I20260927 00:36:52.818732 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.815170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:38845 (local address 127.10.11.194:54815)
0927 00:36:52.815325 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.815344 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.815462 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.815923 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.815926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.815946 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:52.816061 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.816066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.816786 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.816790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.817592 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.817600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.818307 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.818311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.818423 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.818447 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.818532 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.818600 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":19}
I20260927 00:36:52.818732 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.815274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:54815 (local address 127.10.11.195:38845)
0927 00:36:52.815413 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.815417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.815433 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:36:52.815758 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.815762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.815812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.815923 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:52.816167 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.816674 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.817715 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.818196 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.818318 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.818370 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.818547 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.818615 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:36:52.819159 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.815625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:55171 (local address 127.10.11.193:45797)
0927 00:36:52.815753 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.815757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.815910 (+ 153us) server_negotiation.cc:407] Connection header received
0927 00:36:52.816059 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.816062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.816124 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.816216 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:36:52.816488 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.817032 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.818064 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.818557 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.818684 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.818829 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.818918 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.819013 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:36:52.819388 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0562fe88b554763bca3af8f05e9973a" candidate_uuid: "f1b492e2690249128d5f4a1481346313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" is_pre_election: true
I20260927 00:36:52.819509 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.815518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45797 (local address 127.10.11.194:55171)
0927 00:36:52.815809 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.815821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:36:52.815941 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:36:52.816235 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:36:52.816238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:36:52.816253 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:36:52.816373 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.816379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.817141 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.817147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.817936 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:36:52.817944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.818684 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.818688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:36:52.818788 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.818808 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:36:52.819297 (+ 489us) client_negotiation.cc:240] Negotiation successful
0927 00:36:52.819366 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":19,"thread_start_us":127,"threads_started":1}
I20260927 00:36:52.819691 26611 raft_consensus.cc:2454] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b492e2690249128d5f4a1481346313 in term 0.
I20260927 00:36:52.820060 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0562fe88b554763bca3af8f05e9973a" candidate_uuid: "f1b492e2690249128d5f4a1481346313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc91bb392ef74817b4b5d56d8769bfad" is_pre_election: true
I20260927 00:36:52.820175 26418 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313; no voters:
I20260927 00:36:52.820322 26344 raft_consensus.cc:2454] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b492e2690249128d5f4a1481346313 in term 0.
I20260927 00:36:52.820466 26679 raft_consensus.cc:2781] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:36:52.820564 26679 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:36:52.820622 26679 raft_consensus.cc:3037] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:52.821617 26679 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.821977 26679 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [CANDIDATE]: Term 1 election: Requested vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797)
I20260927 00:36:52.822368 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0562fe88b554763bca3af8f05e9973a" candidate_uuid: "f1b492e2690249128d5f4a1481346313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc91bb392ef74817b4b5d56d8769bfad"
I20260927 00:36:52.822371 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0562fe88b554763bca3af8f05e9973a" candidate_uuid: "f1b492e2690249128d5f4a1481346313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
I20260927 00:36:52.822480 26611 raft_consensus.cc:3037] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:52.822507 26344 raft_consensus.cc:3037] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:36:52.823441 26344 raft_consensus.cc:2454] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b492e2690249128d5f4a1481346313 in term 1.
I20260927 00:36:52.823493 26611 raft_consensus.cc:2454] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b492e2690249128d5f4a1481346313 in term 1.
I20260927 00:36:52.823701 26418 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad, f1b492e2690249128d5f4a1481346313; no voters:
I20260927 00:36:52.823868 26679 raft_consensus.cc:2781] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:36:52.824062 26679 raft_consensus.cc:686] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 1 LEADER]: Becoming Leader. State: Replica: f1b492e2690249128d5f4a1481346313, State: Running, Role: LEADER
I20260927 00:36:52.824394 26679 consensus_queue.cc:237] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:36:52.827340 26203 catalog_manager.cc:5495] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1b492e2690249128d5f4a1481346313 (127.10.11.194). New cstate: current_term: 1 leader_uuid: "f1b492e2690249128d5f4a1481346313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } health_report { overall_health: UNKNOWN } } }
I20260927 00:36:52.856575 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.853166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.10.11.194:36515)
0927 00:36:52.853291 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:36:52.853295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:36:52.853308 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:36:52.853415 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:36:52.853417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:36:52.853471 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:36:52.853557 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:36:52.853746 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.854377 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.855164 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:36:52.855368 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:36:52.855454 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:36:52.855762 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:36:52.856179 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:36:52.856267 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:36:52.856406 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:36:52.856455 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:36:52.862347 26344 raft_consensus.cc:1261] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Refusing update from remote peer f1b492e2690249128d5f4a1481346313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:52.862423 26611 raft_consensus.cc:1261] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Refusing update from remote peer f1b492e2690249128d5f4a1481346313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:36:52.863056 26681 consensus_queue.cc:1035] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:52.863265 26679 consensus_queue.cc:1035] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:36:52.877344 26202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:52.877526 26202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:36:52.879837 26202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Sending DeleteTablet for 3 replicas of tablet c0562fe88b554763bca3af8f05e9973a
I20260927 00:36:52.880477 26589 tablet_service.cc:1514] Processing DeleteTablet for tablet c0562fe88b554763bca3af8f05e9973a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:52 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:60680
I20260927 00:36:52.880612 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26241
I20260927 00:36:52.880735 26463 tablet_service.cc:1514] Processing DeleteTablet for tablet c0562fe88b554763bca3af8f05e9973a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:36:52 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:55182
I20260927 00:36:52.880946 26694 tablet_replica.cc:330] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: stopping tablet replica
I20260927 00:36:52.881222 26694 raft_consensus.cc:2229] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:36:52.881451 26695 tablet_replica.cc:330] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: stopping tablet replica
I20260927 00:36:52.881506 26694 raft_consensus.cc:2258] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:52.881803 26695 raft_consensus.cc:2229] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:36:52.882225 26695 raft_consensus.cc:2258] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:36:52.883378 26694 ts_tablet_manager.cc:1905] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:52.883387 26695 ts_tablet_manager.cc:1905] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:36:52.885973 26694 ts_tablet_manager.cc:1918] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:52.885973 26695 ts_tablet_manager.cc:1918] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:36:52.886071 26694 log.cc:1196] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/c0562fe88b554763bca3af8f05e9973a
I20260927 00:36:52.886212 26695 log.cc:1196] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/c0562fe88b554763bca3af8f05e9973a
I20260927 00:36:52.886441 26694 ts_tablet_manager.cc:1939] T c0562fe88b554763bca3af8f05e9973a P 6115eae9ccfc424894efd34a9bc3d6ac: Deleting consensus metadata
I20260927 00:36:52.886670 26695 ts_tablet_manager.cc:1939] T c0562fe88b554763bca3af8f05e9973a P f1b492e2690249128d5f4a1481346313: Deleting consensus metadata
I20260927 00:36:52.887562 26189 catalog_manager.cc:4841] TS f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): tablet c0562fe88b554763bca3af8f05e9973a (table test-table [id=b364fa1793c54a62978632d93ed61aa5]) successfully deleted
I20260927 00:36:52.887755 26189 catalog_manager.cc:4841] TS 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): tablet c0562fe88b554763bca3af8f05e9973a (table test-table [id=b364fa1793c54a62978632d93ed61aa5]) successfully deleted
W20260927 00:36:52.889333 26189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:45797 (error 108)
I20260927 00:36:52.889602 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26395
I20260927 00:36:52.891933 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:36:52.891670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45797 (local address 127.0.0.1:52252)
0927 00:36:52.891792 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:36:52.891859 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:36:52.892069 26189 catalog_manager.cc:4584] TS bc91bb392ef74817b4b5d56d8769bfad (127.10.11.193:45797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0562fe88b554763bca3af8f05e9973a: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45797: connect: Connection refused (error 111)
I20260927 00:36:52.894877 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26532
I20260927 00:36:52.899456 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26170
I20260927 00:37:12.905836 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--webserver_interface=127.10.11.254
--webserver_port=37649
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:13.010957 26697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:13.011183 26697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:13.011220 26697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:13.015053 26697 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:37:13.015093 26697 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:37:13.015108 26697 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:37:13.015122 26697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:13.015139 26697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:13.015156 26697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:13.015172 26697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:13.015187 26697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:13.020058 26697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=37649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:13.020917 26697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:13.021852 26697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:13.030085 26705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.030120 26697 server_base.cc:1034] running on GCE node
W20260927 00:37:13.030138 26702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.030092 26703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.030985 26697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:13.031958 26697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:13.033116 26697 hybrid_clock.cc:648] HybridClock initialized: now 1790469433033107 us; error 25 us; skew 500 ppm
Sep 27 00:37:13 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:13.035727 26697 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:13.036559 26697 webserver.cc:460] Webserver started at http://127.10.11.254:37649/ using document root <none> and password file <none>
I20260927 00:37:13.037083 26697 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:13.037170 26697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:13.040731 26697 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:13.042721 26712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.043877 26697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:13.044016 26697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2c632205c25f44de8844b29edd69d699"
format_stamp: "Formatted at 2026-09-27 00:36:30 on dist-test-slave-6gxc"
server_key: "6f987aa721069972a872364ee91b54f5"
server_key_iv: "d06ffffa82d18984180c037cf2c2a305"
server_key_version: "encryptionkey@0"
I20260927 00:37:13.044525 26697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:13.059441 26697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:13.061936 26697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:13.062176 26697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:13.068655 26697 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:33691
I20260927 00:37:13.068907 26764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:33691 every 8 connection(s)
I20260927 00:37:13.069639 26697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:13.071734 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26697
I20260927 00:37:13.072165 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:45797
--local_ip_for_outbound_sockets=127.10.11.193
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=39379
--webserver_interface=127.10.11.193
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:37:13.073705 26765 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:37:13.075321 26765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap starting.
I20260927 00:37:13.092635 26765 log.cc:824] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:13.103495 26765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:37:13.103911 26765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Bootstrap complete.
I20260927 00:37:13.107877 26765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:37:13.108299 26765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Initialized, Role: FOLLOWER
I20260927 00:37:13.108758 26765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:37:13.108901 26765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:13.108994 26765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:13.109115 26765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:37:13.110956 26765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:37:13.111279 26765 leader_election.cc:302] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c632205c25f44de8844b29edd69d699; no voters:
I20260927 00:37:13.111595 26765 leader_election.cc:288] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:37:13.111671 26769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:37:13.111853 26769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [term 3 LEADER]: Becoming Leader. State: Replica: 2c632205c25f44de8844b29edd69d699, State: Running, Role: LEADER
I20260927 00:37:13.112131 26769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } }
I20260927 00:37:13.112742 26765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:13.113011 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c632205c25f44de8844b29edd69d699. Latest consensus state: current_term: 3 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:37:13.113128 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:13.113458 26773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:13.113611 26770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2c632205c25f44de8844b29edd69d699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c632205c25f44de8844b29edd69d699" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 33691 } } }
I20260927 00:37:13.113817 26770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:13.118384 26773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:13.119055 26773 catalog_manager.cc:1259] Loaded cluster ID: f61ae7db5506448383057761c9bb05dd
I20260927 00:37:13.119119 26773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:13.121531 26773 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:13.122303 26773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Loaded TSK: 0
I20260927 00:37:13.122386 26773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c632205c25f44de8844b29edd69d699: Loaded TSK: 1
F20260927 00:37:13.123102 26773 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f11313c49ad google::LogMessage::Fail() at ??:0
@ 0x7f11313c788a google::LogMessage::SendToLog() at ??:0
@ 0x7f11313c44de google::LogMessage::Flush() at ??:0
@ 0x7f11313c5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f113408818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1134080f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f113bd2d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f113bd0799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f113bd08178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f113bd49df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f113bd6882b std::function<>::operator()() at ??:0
@ 0x7f113bd07b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f113bd08f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f113bd05579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f113bd49640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f11336163d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1133616c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f11336183f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1133607512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1134c4b6db start_thread at ??:0
@ 0x7f112fec261f clone at ??:0
W20260927 00:37:13.208392 26767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:13.208726 26767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:13.208799 26767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:13.215061 26767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:13.215116 26767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:13.215202 26767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:13.223069 26767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:45797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=39379
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:13.224371 26767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:13.225693 26767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:13.236400 26813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.236505 26767 server_base.cc:1034] running on GCE node
W20260927 00:37:13.236377 26812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.236639 26815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.237056 26767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:13.237728 26767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:13.238909 26767 hybrid_clock.cc:648] HybridClock initialized: now 1790469433238896 us; error 27 us; skew 500 ppm
Sep 27 00:37:13 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:13.241631 26767 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:37:13.242573 26767 webserver.cc:460] Webserver started at http://127.10.11.193:39379/ using document root <none> and password file <none>
I20260927 00:37:13.243105 26767 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:13.243191 26767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:13.246920 26767 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:13.249184 26822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.250494 26767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:13.250613 26767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "bc91bb392ef74817b4b5d56d8769bfad"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "e8aae053d6ad89ecbddbbee99095d178"
server_key_iv: "4100296a1d795f229df5cd0e9b21bcd9"
server_key_version: "encryptionkey@0"
I20260927 00:37:13.251153 26767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:13.262264 26767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:13.265100 26767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:13.265277 26767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:13.265831 26767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:13.266949 26829 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:37:13.272037 26767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:37:13.272125 26767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:37:13.272198 26767 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:37:13.273767 26767 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:37:13.273833 26767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:37:13.274135 26829 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap starting.
I20260927 00:37:13.284143 26767 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:45797
I20260927 00:37:13.284349 26936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:45797 every 8 connection(s)
I20260927 00:37:13.285077 26767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:13.288187 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26767
I20260927 00:37:13.288673 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:36515
--local_ip_for_outbound_sockets=127.10.11.194
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=43305
--webserver_interface=127.10.11.194
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:37:13.296451 26829 log.cc:824] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:13.303134 26829 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:37:13.303481 26829 tablet_bootstrap.cc:492] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Bootstrap complete.
I20260927 00:37:13.304729 26829 ts_tablet_manager.cc:1397] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260927 00:37:13.307092 26829 raft_consensus.cc:348] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:13.307483 26829 raft_consensus.cc:729] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc91bb392ef74817b4b5d56d8769bfad, State: Initialized, Role: FOLLOWER
I20260927 00:37:13.307945 26829 consensus_queue.cc:260] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:13.308849 26829 ts_tablet_manager.cc:1428] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 00:37:13.372099 26939 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:37:13.287005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:51815)
0927 00:37:13.287464 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.287515 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:13.288660 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:13.371191 (+ 82531us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
W20260927 00:37:13.372601 26937 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:33691 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:33691: Client connection negotiation failed: client connection to 127.10.11.254:33691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:37:13.373350 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.373055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:45667)
0927 00:37:13.373202 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.373277 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:37:13.374082 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.373846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:44853)
0927 00:37:13.373961 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.374013 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:37:13.374282 26937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:37:13.409324 26940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:13.409541 26940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:13.409581 26940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:13.413358 26940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:13.413403 26940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:13.413455 26940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:13.418819 26940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:36515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=43305
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:13.419651 26940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:13.420511 26940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:13.429775 26948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.429826 26947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.429745 26950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.429905 26940 server_base.cc:1034] running on GCE node
I20260927 00:37:13.430428 26940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:13.430996 26940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:13.432176 26940 hybrid_clock.cc:648] HybridClock initialized: now 1790469433432165 us; error 50 us; skew 500 ppm
Sep 27 00:37:13 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:13.434796 26940 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:37:13.435575 26940 webserver.cc:460] Webserver started at http://127.10.11.194:43305/ using document root <none> and password file <none>
I20260927 00:37:13.436090 26940 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:13.436178 26940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:13.439797 26940 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:13.441747 26957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.443011 26940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:13.443135 26940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "f1b492e2690249128d5f4a1481346313"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "517557fec4e3f65442c77ad7f201a8d7"
server_key_iv: "4e6a660239a5192066b4cd988b3f4d67"
server_key_version: "encryptionkey@0"
I20260927 00:37:13.443634 26940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:13.458451 26940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:13.461587 26940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:13.461777 26940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:13.462412 26940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:13.463160 26940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:13.463229 26940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.463297 26940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:13.463352 26940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.472072 26940 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:36515
I20260927 00:37:13.472119 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:36515 every 8 connection(s)
I20260927 00:37:13.473031 26940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:13.476509 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.475112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:37083)
0927 00:37:13.475756 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.475861 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":106,"threads_started":1}
W20260927 00:37:13.476922 27071 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:33691 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:33691: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
I20260927 00:37:13.477519 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.477285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:43187)
0927 00:37:13.477409 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.477466 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:37:13.478188 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.477920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:41365)
0927 00:37:13.478025 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.478073 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:37:13.478403 27071 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:37:13.478407 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 26940
I20260927 00:37:13.478838 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:38845
--local_ip_for_outbound_sockets=127.10.11.195
--tserver_master_addrs=127.10.11.254:33691
--webserver_port=36095
--webserver_interface=127.10.11.195
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:13.585731 27074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:13.585949 27074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:13.585988 27074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:13.589911 27074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:13.589962 27074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:13.590044 27074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:13.594985 27074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:38845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=36095
--tserver_master_addrs=127.10.11.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:13.595911 27074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:13.596825 27074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:13.605937 27080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.605968 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:13.606027 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:13.606626 27074 server_base.cc:1034] running on GCE node
I20260927 00:37:13.607019 27074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:13.607676 27074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:13.608871 27074 hybrid_clock.cc:648] HybridClock initialized: now 1790469433608863 us; error 30 us; skew 500 ppm
Sep 27 00:37:13 dist-test-slave-6gxc krb5kdc[25623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:13.611682 27074 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:37:13.612533 27074 webserver.cc:460] Webserver started at http://127.10.11.195:36095/ using document root <none> and password file <none>
I20260927 00:37:13.613072 27074 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:13.613163 27074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:13.616784 27074 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:13.618806 27089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.619930 27074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:13.620087 27074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "6115eae9ccfc424894efd34a9bc3d6ac"
format_stamp: "Formatted at 2026-09-27 00:36:31 on dist-test-slave-6gxc"
server_key: "bba18d08ad1d4f0425ff1576cc6b81f3"
server_key_iv: "736aedffbdb5d39d0d6395e3b05bbcac"
server_key_version: "encryptionkey@0"
I20260927 00:37:13.620599 27074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:13.635605 27074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:13.638274 27074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:13.638453 27074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:13.638999 27074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:13.639770 27074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:13.639840 27074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.639911 27074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:13.639963 27074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:13.648871 27074 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:38845
I20260927 00:37:13.648890 27202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:38845 every 8 connection(s)
I20260927 00:37:13.649817 27074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:13.653211 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.651856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:53099)
0927 00:37:13.652469 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.652571 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":107,"threads_started":1}
W20260927 00:37:13.653653 27203 heartbeater.cc:644] Failed to heartbeat to 127.10.11.254:33691 (0 consecutive failures): Network error: Failed to ping master at 127.10.11.254:33691: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
I20260927 00:37:13.654306 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27074
I20260927 00:37:13.654769 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.653965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:47167)
0927 00:37:13.654677 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.654721 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:37:13.655484 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:13.655263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:56409)
0927 00:37:13.655370 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:13.655422 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:37:13.655669 27203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:37:14.375190 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:14.374801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:44011)
0927 00:37:14.375032 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:14.375083 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260927 00:37:14.479327 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:14.478960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:53473)
0927 00:37:14.479188 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:14.479238 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
I20260927 00:37:14.656481 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:14.656131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:42375)
0927 00:37:14.656343 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:14.656390 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260927 00:37:15.097204 27210 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:15.097384 27210 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:15.098441 27210 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
Sep 27 00:37:15 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.195@KRBTEST.COM
Sep 27 00:37:15 dist-test-slave-6gxc krb5kdc[25623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.194@KRBTEST.COM
I20260927 00:37:15.115705 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.098685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:38845 (local address 127.10.11.193:34201)
0927 00:37:15.099268 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:15.099283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:15.099402 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:15.101347 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:15.101353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:15.102014 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:15.102736 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:15.102756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.104659 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.104662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:15.106315 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:15.106326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.107244 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.107248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:15.107523 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:15.108273 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:15.108314 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:15.110232 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:15.112723 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:15.112730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:15.112746 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:15.113200 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:15.113594 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:15.113599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:15.113802 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:15.114557 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:15.114559 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:15.114893 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 00:37:15.115106 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:37:15.115519 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":478,"spinlock_wait_cycles":71168,"thread_start_us":78,"threads_started":1}
I20260927 00:37:15.115727 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.098848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:34201 (local address 127.10.11.195:38845)
0927 00:37:15.099292 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:15.099323 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:15.099371 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:37:15.100405 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:15.100413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:15.100902 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:15.101238 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:15.102918 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.104504 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.106433 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.107122 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.107283 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:15.110464 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:15.110469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:15.110480 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:15.110512 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:15.112526 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113326 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:15.113332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:15.113337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:15.113502 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113908 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:15.113911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:15.113913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:15.114422 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:15.114575 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:15.115274 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 00:37:15.115559 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":124,"threads_started":1}
I20260927 00:37:15.115705 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.099024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:36515 (local address 127.10.11.193:48119)
0927 00:37:15.099268 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:15.099283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:15.099402 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:15.101347 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:15.101353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:15.101927 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:15.102736 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:15.102756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.104659 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.104662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:15.106318 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:15.106326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.107244 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.107248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:15.107527 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:15.108278 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:15.108300 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:15.110582 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:15.112754 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:15.112756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:15.112758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:15.113200 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:15.113594 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:15.113599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:15.113802 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:15.114548 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:15.114554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:15.114893 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 00:37:15.115106 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:37:15.115469 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":452,"spinlock_wait_cycles":51584,"thread_start_us":60,"threads_started":1}
I20260927 00:37:15.115727 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.099182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:48119 (local address 127.10.11.194:36515)
0927 00:37:15.099587 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:15.099600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:15.099636 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:37:15.100405 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:15.100413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:15.100902 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:15.101238 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:15.102918 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.104504 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.106433 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:15.107122 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:15.107283 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:15.110759 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:15.110766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:15.110781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:15.110817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:15.112677 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113326 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:15.113332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:15.113337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:15.113495 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:15.113908 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:15.113911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:15.113913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:15.114422 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:15.114559 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:15.115290 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 00:37:15.115551 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":103,"threads_started":1}
W20260927 00:37:15.118261 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
W20260927 00:37:15.118471 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
I20260927 00:37:15.118530 26825 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad; no voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313
I20260927 00:37:15.118809 27210 raft_consensus.cc:2726] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:37:15.376259 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.375984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:33223)
0927 00:37:15.376127 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:15.376180 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:37:15.480413 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.480031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:46647)
0927 00:37:15.480260 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:15.480310 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260927 00:37:15.657466 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:15.657115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:58187)
0927 00:37:15.657334 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:15.657385 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260927 00:37:16.377348 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:16.376946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:49033)
0927 00:37:16.377208 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:16.377259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260927 00:37:16.481465 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:16.481042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:51149)
0927 00:37:16.481317 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:16.481388 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":64,"threads_started":1}
I20260927 00:37:16.658640 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:16.658215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:52579)
0927 00:37:16.658487 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:16.658549 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260927 00:37:16.854048 27220 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:16.854246 27220 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:16.854646 27220 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
W20260927 00:37:16.855559 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
W20260927 00:37:16.855710 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
I20260927 00:37:16.855753 26825 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad; no voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313
I20260927 00:37:16.855888 27220 raft_consensus.cc:2726] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:37:17.378432 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:17.378063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:54949)
0927 00:37:17.378275 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:17.378324 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260927 00:37:17.482534 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:17.482167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:54853)
0927 00:37:17.482389 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:17.482439 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260927 00:37:17.659632 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:17.659280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:47561)
0927 00:37:17.659500 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:17.659550 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 00:37:18.379698 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:18.379270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:51463)
0927 00:37:18.379556 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:18.379608 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260927 00:37:18.483606 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:18.483188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:59115)
0927 00:37:18.483425 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:18.483478 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260927 00:37:18.660827 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:18.660457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:47507)
0927 00:37:18.660688 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:18.660741 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260927 00:37:19.380836 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:19.380468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:48291)
0927 00:37:19.380695 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:19.380744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260927 00:37:19.402473 27228 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:19.402599 27228 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:19.402994 27228 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
W20260927 00:37:19.403860 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
W20260927 00:37:19.404003 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
I20260927 00:37:19.404062 26825 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad; no voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313
I20260927 00:37:19.404212 27228 raft_consensus.cc:2726] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:37:19.484604 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:19.484231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:34419)
0927 00:37:19.484475 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:19.484523 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260927 00:37:19.662040 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:19.661611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:58315)
0927 00:37:19.661883 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:19.661936 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260927 00:37:20.381906 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:20.381497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:34759)
0927 00:37:20.381763 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:20.381814 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260927 00:37:20.485723 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:20.485266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:58059)
0927 00:37:20.485588 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:20.485640 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260927 00:37:20.663332 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:20.662828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:49601)
0927 00:37:20.663174 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:20.663239 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":84,"threads_started":1}
I20260927 00:37:21.252164 27234 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:21.252307 27234 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:21.252722 27234 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
W20260927 00:37:21.253561 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
W20260927 00:37:21.253696 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
I20260927 00:37:21.253753 26825 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad; no voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313
I20260927 00:37:21.253969 27234 raft_consensus.cc:2726] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:37:21.383297 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:21.382842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:58133)
0927 00:37:21.383134 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:21.383202 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260927 00:37:21.486924 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:21.486523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:49741)
0927 00:37:21.486769 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:21.486829 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260927 00:37:21.664431 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:21.664064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:42377)
0927 00:37:21.664293 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:21.664345 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260927 00:37:22.384554 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:22.384154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:51043)
0927 00:37:22.384401 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:22.384461 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260927 00:37:22.488006 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:22.487574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:46113)
0927 00:37:22.487863 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:22.487926 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260927 00:37:22.665575 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:22.665198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:49475)
0927 00:37:22.665438 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:22.665489 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260927 00:37:23.385857 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:23.385359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:45977)
0927 00:37:23.385669 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:23.385740 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260927 00:37:23.489084 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:23.488680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:36637)
0927 00:37:23.488943 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:23.488997 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260927 00:37:23.666648 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:23.666287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:52011)
0927 00:37:23.666509 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:23.666559 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260927 00:37:24.386924 27244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:24.386521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:35443)
0927 00:37:24.386768 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:24.386829 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
I20260927 00:37:24.490114 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:24.489726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:42479)
0927 00:37:24.489958 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:24.490004 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260927 00:37:24.667670 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:24.667287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:38065)
0927 00:37:24.667521 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:24.667579 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260927 00:37:25.387979 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:25.387610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:46547)
0927 00:37:25.387829 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:25.387877 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260927 00:37:25.491137 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:25.490775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:54533)
0927 00:37:25.491001 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:25.491054 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260927 00:37:25.668840 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:25.668456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:36169)
0927 00:37:25.668692 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:25.668751 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260927 00:37:26.389108 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:26.388711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:49035)
0927 00:37:26.388962 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:26.389016 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260927 00:37:26.492532 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:26.491833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:40863)
0927 00:37:26.492254 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:26.492429 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":81,"threads_started":1}
I20260927 00:37:26.670702 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:26.669750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:38593)
0927 00:37:26.670461 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:26.670568 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":92,"threads_started":1}
I20260927 00:37:27.390224 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:27.389815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:55675)
0927 00:37:27.390058 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:27.390132 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260927 00:37:27.493750 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:27.493354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:38649)
0927 00:37:27.493610 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:27.493663 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260927 00:37:27.671994 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:27.671578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.195:38425)
0927 00:37:27.671857 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:27.671907 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260927 00:37:28.095849 27256 raft_consensus.cc:482] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:28.095975 27256 raft_consensus.cc:504] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6115eae9ccfc424894efd34a9bc3d6ac" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 38845 } } peers { permanent_uuid: "f1b492e2690249128d5f4a1481346313" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 36515 } } peers { permanent_uuid: "bc91bb392ef74817b4b5d56d8769bfad" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45797 } }
I20260927 00:37:28.096448 27256 leader_election.cc:288] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845), f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515)
W20260927 00:37:28.097189 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115eae9ccfc424894efd34a9bc3d6ac (127.10.11.195:38845): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
W20260927 00:37:28.097337 26825 leader_election.cc:341] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1b492e2690249128d5f4a1481346313 (127.10.11.194:36515): Not found: Tablet not found: c0562fe88b554763bca3af8f05e9973a
I20260927 00:37:28.097386 26825 leader_election.cc:302] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc91bb392ef74817b4b5d56d8769bfad; no voters: 6115eae9ccfc424894efd34a9bc3d6ac, f1b492e2690249128d5f4a1481346313
I20260927 00:37:28.097509 27256 raft_consensus.cc:2726] T c0562fe88b554763bca3af8f05e9973a P bc91bb392ef74817b4b5d56d8769bfad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:37:28.391826 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:28.391407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.193:46943)
0927 00:37:28.391683 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:28.391737 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":117,"threads_started":1}
I20260927 00:37:28.495014 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:28.494606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:33691 (local address 127.10.11.194:49983)
0927 00:37:28.494860 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:28.494912 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.254:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260927 00:37:28.660040 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26767
I20260927 00:37:28.665907 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 26940
I20260927 00:37:28.671260 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27074
2026-09-27T00:37:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59092 ms)
[----------] 30 tests from SecurityITest (235234 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:28 dist-test-slave-6gxc krb5kdc[27261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:28 dist-test-slave-6gxc krb5kdc[27261](info): set up 2 sockets
Sep 27 00:37:28 dist-test-slave-6gxc krb5kdc[27261](info): commencing operation
krb5kdc: starting...
W20260927 00:37:29.709813 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:29 dist-test-slave-6gxc krb5kdc[27261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:29.830801 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43829
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:29.943768 27274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:29.944058 27274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:29.944142 27274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:29.948041 27274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:29.948146 27274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:29.948196 27274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:29.948233 27274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:29.948267 27274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:29.953280 27274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:29.954269 27274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:29.955433 27274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:29.963728 27282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:29.963773 27279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:29.963740 27280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:29.963794 27274 server_base.cc:1034] running on GCE node
I20260927 00:37:29.964643 27274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:29.965639 27274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:29.966810 27274 hybrid_clock.cc:648] HybridClock initialized: now 1790469449966779 us; error 40 us; skew 500 ppm
Sep 27 00:37:29 dist-test-slave-6gxc krb5kdc[27261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469449, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:29.969553 27274 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:29.970500 27274 webserver.cc:460] Webserver started at http://127.10.11.254:36879/ using document root <none> and password file <none>
I20260927 00:37:29.971050 27274 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:29.971143 27274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:29.971377 27274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:29.973345 27274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "82b0cc560dd04dc6ba68ea3e84132354"
format_stamp: "Formatted at 2026-09-27 00:37:29 on dist-test-slave-6gxc"
server_key: "e8b313b1e675371df39a335ea17c3690"
server_key_iv: "6ca40032eb47cbf7f1de54dab262f3d7"
server_key_version: "encryptionkey@0"
I20260927 00:37:29.973948 27274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "82b0cc560dd04dc6ba68ea3e84132354"
format_stamp: "Formatted at 2026-09-27 00:37:29 on dist-test-slave-6gxc"
server_key: "e8b313b1e675371df39a335ea17c3690"
server_key_iv: "6ca40032eb47cbf7f1de54dab262f3d7"
server_key_version: "encryptionkey@0"
I20260927 00:37:29.977574 27274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:37:29.980000 27289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:29.981155 27274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:29.981299 27274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "82b0cc560dd04dc6ba68ea3e84132354"
format_stamp: "Formatted at 2026-09-27 00:37:29 on dist-test-slave-6gxc"
server_key: "e8b313b1e675371df39a335ea17c3690"
server_key_iv: "6ca40032eb47cbf7f1de54dab262f3d7"
server_key_version: "encryptionkey@0"
I20260927 00:37:29.981415 27274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:30.008939 27274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:30.011921 27274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:30.012218 27274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:30.020323 27274 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:43829
I20260927 00:37:30.020349 27341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:43829 every 8 connection(s)
I20260927 00:37:30.021260 27274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:30.024125 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:30.026657 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27274
I20260927 00:37:30.026794 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:30.027117 10287 external_mini_cluster.cc:1428] Setting key c299399bcc5f1d37d9b019748b561cba
I20260927 00:37:30.030476 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: Bootstrap starting.
I20260927 00:37:30.032634 27342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:30.033344 27342 log.cc:824] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:30.035501 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: No bootstrap required, opened a new log
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:30.038784 27342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } }
I20260927 00:37:30.039026 27342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:30.039109 27342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b0cc560dd04dc6ba68ea3e84132354, State: Initialized, Role: FOLLOWER
I20260927 00:37:30.039595 27342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } }
I20260927 00:37:30.039743 27342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:30.039810 27342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:30.039906 27342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:30.040766 27342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } }
I20260927 00:37:30.041097 27342 leader_election.cc:302] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82b0cc560dd04dc6ba68ea3e84132354; no voters:
I20260927 00:37:30.041427 27342 leader_election.cc:288] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:30.041499 27347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:30.041674 27347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [term 1 LEADER]: Becoming Leader. State: Replica: 82b0cc560dd04dc6ba68ea3e84132354, State: Running, Role: LEADER
I20260927 00:37:30.042032 27347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } }
I20260927 00:37:30.042615 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.028692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.10.11.254:43829)
0927 00:37:30.029116 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.029130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.029165 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:37:30.029802 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.029820 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.030234 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.030596 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:30.031369 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.032324 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.033230 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.033782 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.034188 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.036668 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:30.036673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:30.036685 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:30.036710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:30.039326 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.039764 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.039770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.039776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.039912 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.040223 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.040228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.040230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.040611 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:30.040818 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.041377 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.041687 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260927 00:37:30.042680 27342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:30.043685 27348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b0cc560dd04dc6ba68ea3e84132354. Latest consensus state: current_term: 1 leader_uuid: "82b0cc560dd04dc6ba68ea3e84132354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } } }
I20260927 00:37:30.043803 27348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:30.044080 27349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82b0cc560dd04dc6ba68ea3e84132354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b0cc560dd04dc6ba68ea3e84132354" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43829 } } }
I20260927 00:37:30.044153 27349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:30.044950 27356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:30.047646 27356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:30.052698 27356 catalog_manager.cc:1347] Generated new cluster ID: d4d15994e54d49cdb3f51156db1ea638
I20260927 00:37:30.052762 27356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:30.059831 27356 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:30.061275 27356 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:30.070864 27356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: Generated new TSK 0
I20260927 00:37:30.071624 27356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:37:30.101279 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:30.206535 27368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:30.206763 27368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:30.206804 27368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:30.210649 27368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:30.210698 27368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:30.210752 27368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:30.215610 27368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:30.216614 27368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:30.217545 27368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:30.226574 27374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.226547 27373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.226557 27376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:30.226739 27368 server_base.cc:1034] running on GCE node
I20260927 00:37:30.227201 27368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:30.227820 27368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:30.229003 27368 hybrid_clock.cc:648] HybridClock initialized: now 1790469450228982 us; error 35 us; skew 500 ppm
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:30.231597 27368 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:37:30.232417 27368 webserver.cc:460] Webserver started at http://127.10.11.193:42423/ using document root <none> and password file <none>
I20260927 00:37:30.232940 27368 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:30.233052 27368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:30.233260 27368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:30.235210 27368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "9fcff2a6d59744f3abc8d5363fdd2663"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "75633a25132959590c7573ec21a8403b"
server_key_iv: "3e453e06cb8a8fa4c0b2da46c2bc8c47"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.235737 27368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "9fcff2a6d59744f3abc8d5363fdd2663"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "75633a25132959590c7573ec21a8403b"
server_key_iv: "3e453e06cb8a8fa4c0b2da46c2bc8c47"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.239114 27368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 00:37:30.241294 27383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.242477 27368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:30.242601 27368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "9fcff2a6d59744f3abc8d5363fdd2663"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "75633a25132959590c7573ec21a8403b"
server_key_iv: "3e453e06cb8a8fa4c0b2da46c2bc8c47"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.242702 27368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:30.260852 27368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:30.263543 27368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:30.263736 27368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:30.264284 27368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:30.265050 27368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:30.265120 27368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.265188 27368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:30.265230 27368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.274060 27368 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44599
I20260927 00:37:30.274083 27496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44599 every 8 connection(s)
I20260927 00:37:30.274887 27368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:30.276630 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27368
I20260927 00:37:30.276738 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:30.277097 10287 external_mini_cluster.cc:1428] Setting key 5f49100f39037373265f59c60b826a11
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:37:30.290972 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.276939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:44327 (local address 127.10.11.254:43829)
0927 00:37:30.277091 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.277097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.278069 (+ 972us) server_negotiation.cc:407] Connection header received
0927 00:37:30.279924 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.279927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.279972 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.280055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:30.281390 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.282043 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.284068 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.284353 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.284476 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.287349 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:30.287352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:30.287357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:30.287384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:30.288783 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.289292 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.289298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.289300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.289377 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.289709 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.289714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.289715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.289969 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:30.290074 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.290679 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.290801 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:37:30.291986 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.277327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43829 (local address 127.10.11.193:44327)
0927 00:37:30.277925 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.277967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.279597 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.280258 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.280269 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.280762 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:30.281221 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.281235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.282244 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.282251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.283897 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.283919 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.284542 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.284550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.284800 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.285500 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:30.285520 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:30.287187 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:30.288903 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.288908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.288921 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.289187 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.289469 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.289474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.289476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.289614 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.290091 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:30.290121 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:30.290403 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.290608 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.290916 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":173,"threads_started":1}
I20260927 00:37:30.293272 27497 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43829
I20260927 00:37:30.293519 27497 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:30.294167 27497 heartbeater.cc:511] Master 127.10.11.254:43829 requested a full tablet report, sending...
I20260927 00:37:30.296097 27306 ts_manager.cc:194] Registered new tserver with Master: 9fcff2a6d59744f3abc8d5363fdd2663 (127.10.11.193:44599)
I20260927 00:37:30.297580 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:44327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:37:30.307616 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:30.414852 27502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:30.415072 27502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:30.415282 27502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:30.419523 27502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:30.419579 27502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:30.419664 27502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:30.424693 27502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:30.425597 27502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:30.426530 27502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:30.435575 27507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.435603 27510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.435570 27508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:30.435904 27502 server_base.cc:1034] running on GCE node
I20260927 00:37:30.436252 27502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:30.436857 27502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:30.438041 27502 hybrid_clock.cc:648] HybridClock initialized: now 1790469450438013 us; error 47 us; skew 500 ppm
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:30.440780 27502 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:37:30.441565 27502 webserver.cc:460] Webserver started at http://127.10.11.194:33413/ using document root <none> and password file <none>
I20260927 00:37:30.442082 27502 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:30.442238 27502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:30.442480 27502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:30.444860 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "a9263738745e267967876e78456d8517"
server_key_iv: "d05e18016dfe29efacb19d64814f1988"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.445355 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "a9263738745e267967876e78456d8517"
server_key_iv: "d05e18016dfe29efacb19d64814f1988"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.448808 27502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:37:30.450965 27517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.452169 27502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:30.452265 27502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "a9263738745e267967876e78456d8517"
server_key_iv: "d05e18016dfe29efacb19d64814f1988"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.452384 27502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:30.464255 27502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:30.466775 27502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:30.466951 27502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:30.467458 27502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:30.468192 27502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:30.468237 27502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.468302 27502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:30.468344 27502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.476536 27502 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:40537
I20260927 00:37:30.476581 27630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:40537 every 8 connection(s)
I20260927 00:37:30.477327 27502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:30.482895 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27502
I20260927 00:37:30.483024 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:37:30.483333 10287 external_mini_cluster.cc:1428] Setting key 830c1d125e740c534dad44526f47af3d
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:30.491081 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.479022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:43855 (local address 127.10.11.254:43829)
0927 00:37:30.479144 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.479148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.479892 (+ 744us) server_negotiation.cc:407] Connection header received
0927 00:37:30.481156 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.481159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.481216 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.481316 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:30.482402 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.483154 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.484735 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.485043 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.485156 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.487582 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:30.487590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:30.487592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:30.487614 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:30.489114 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.489595 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.489601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.489602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.489681 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.490006 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.490010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.490012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.490250 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:30.490354 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.490805 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.490918 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:37:30.491858 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.479335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43829 (local address 127.10.11.194:43855)
0927 00:37:30.479762 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.479801 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.480953 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.481476 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.481483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.481790 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:30.482269 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.482283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.483290 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.483293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.484560 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.484570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.485197 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.485200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.485329 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.485955 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:30.485971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:30.487422 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:30.489227 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.489233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.489245 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.489492 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.489778 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.489781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.489783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.489913 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.490352 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:30.490359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:30.490585 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.490761 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.491057 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:37:30.492960 27631 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43829
I20260927 00:37:30.493203 27631 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:30.493600 27631 heartbeater.cc:511] Master 127.10.11.254:43829 requested a full tablet report, sending...
I20260927 00:37:30.494629 27306 ts_manager.cc:194] Registered new tserver with Master: 3ec6bc2f2be84e40b5ad4be3e20d05ab (127.10.11.194:40537)
I20260927 00:37:30.495246 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:43855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:37:30.509477 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:30.614800 27636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:30.615052 27636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:30.615126 27636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:30.619014 27636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:30.619093 27636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:30.619179 27636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:30.624305 27636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:30.625267 27636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:30.626245 27636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:30.635016 27644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.635016 27641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:30.635046 27642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:30.635263 27636 server_base.cc:1034] running on GCE node
I20260927 00:37:30.635692 27636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:30.636255 27636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:30.637437 27636 hybrid_clock.cc:648] HybridClock initialized: now 1790469450637427 us; error 39 us; skew 500 ppm
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:30.639982 27636 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:37:30.640686 27636 webserver.cc:460] Webserver started at http://127.10.11.195:34499/ using document root <none> and password file <none>
I20260927 00:37:30.641145 27636 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:30.641201 27636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:30.641342 27636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:30.643299 27636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "24d6f8d7fa4b4491a01feda14c727bc8"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "8d8d644bdeb58de268feab0de8896cb6"
server_key_iv: "1b267e1c79be165c90d4e9bed0ff53db"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.643834 27636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "24d6f8d7fa4b4491a01feda14c727bc8"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "8d8d644bdeb58de268feab0de8896cb6"
server_key_iv: "1b267e1c79be165c90d4e9bed0ff53db"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.647471 27636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:30.649580 27651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.650745 27636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:30.650830 27636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "24d6f8d7fa4b4491a01feda14c727bc8"
format_stamp: "Formatted at 2026-09-27 00:37:30 on dist-test-slave-6gxc"
server_key: "8d8d644bdeb58de268feab0de8896cb6"
server_key_iv: "1b267e1c79be165c90d4e9bed0ff53db"
server_key_version: "encryptionkey@0"
I20260927 00:37:30.650903 27636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:30.673920 27636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:30.676550 27636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:30.676687 27636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:30.677152 27636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:30.677850 27636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:30.677893 27636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.677932 27636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:30.677946 27636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:30.686705 27636 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:40651
I20260927 00:37:30.686728 27764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:40651 every 8 connection(s)
I20260927 00:37:30.687528 27636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:30.695012 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27636
I20260927 00:37:30.695154 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:30.695566 10287 external_mini_cluster.cc:1428] Setting key a7a74e61f49fa7c842d48127c2a3469c
Sep 27 00:37:30 dist-test-slave-6gxc krb5kdc[27261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:30.700758 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.689221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:51387 (local address 127.10.11.254:43829)
0927 00:37:30.689357 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.689361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.690155 (+ 794us) server_negotiation.cc:407] Connection header received
0927 00:37:30.691466 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.691469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.691524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.691604 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:30.692657 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.693237 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.694375 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.694597 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.694707 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.697464 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:30.697467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:30.697469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:30.697491 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:30.699024 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.699472 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.699483 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.699485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.699564 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.699892 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.699894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.699895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.700061 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:30.700140 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.700556 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.700651 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:37:30.701467 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.689591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43829 (local address 127.10.11.195:51387)
0927 00:37:30.689999 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.690042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.691263 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.691760 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.691767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.692081 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:30.692504 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.692526 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.693357 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.693360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.694252 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.694260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.694738 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.694741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.694901 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.695525 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:30.695540 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:30.697295 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:30.699122 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.699126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.699136 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.699364 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.699649 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:30.699651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:30.699652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:30.699767 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:30.700151 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:30.700155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:30.700347 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.700487 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.700700 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260927 00:37:30.702600 27765 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43829
I20260927 00:37:30.702807 27765 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:30.703224 27765 heartbeater.cc:511] Master 127.10.11.254:43829 requested a full tablet report, sending...
I20260927 00:37:30.704159 27306 ts_manager.cc:194] Registered new tserver with Master: 24d6f8d7fa4b4491a01feda14c727bc8 (127.10.11.195:40651)
I20260927 00:37:30.704696 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:51387
I20260927 00:37:30.709240 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:30.717280 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.711047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.10.11.254:43829)
0927 00:37:30.711153 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.711156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.711358 (+ 202us) server_negotiation.cc:407] Connection header received
0927 00:37:30.711449 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.711450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.711482 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.711549 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:30.712141 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.712594 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.713369 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.713568 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.713637 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.714808 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:30.714811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:30.714813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:30.714832 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:30.716000 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.716349 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.716352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.716353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.716415 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:30.716637 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:30.716639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:30.716640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:30.716805 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:30.716899 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.717106 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.717185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:37:30.719728 27306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:30.722594 27306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:30.735459 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.730342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44599 (local address 127.0.0.1:57468)
0927 00:37:30.730605 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.730617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.730698 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.731572 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.731575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.731592 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:30.731702 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.731710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.733024 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.733027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.733819 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.733828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.734914 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.734917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.735031 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.735067 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.735197 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.735301 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260927 00:37:30.735620 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.729916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:40651 (local address 127.0.0.1:45896)
0927 00:37:30.730265 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.730309 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.730421 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.731172 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.731178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.731200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:30.731306 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.731312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.732821 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.732824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.734148 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.734156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.735208 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.735212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.735291 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.735310 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.735391 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.735446 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":25,"thread_start_us":78,"threads_started":1}
I20260927 00:37:30.735790 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.730045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.10.11.195:40651)
0927 00:37:30.730702 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.730713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.730735 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:37:30.730800 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.730805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.730991 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.731450 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:30.731578 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.732700 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.734273 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.735087 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.735195 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.735307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.735417 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.735519 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":152,"threads_started":1}
I20260927 00:37:30.736135 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.730753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.10.11.193:44599)
0927 00:37:30.731113 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.731127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.731151 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:37:30.731216 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.731221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.731394 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.731576 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:30.731819 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.732895 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.733979 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.734795 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.734915 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.735062 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.735905 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.736020 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":134,"threads_started":1}
I20260927 00:37:30.737051 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.730609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:40537 (local address 127.0.0.1:37888)
0927 00:37:30.731936 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:30.731946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:30.732012 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:30.732531 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:30.732534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:30.732551 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:30.732630 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.732636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.734056 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.734059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.735402 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:30.735411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.736655 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.736659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:30.736740 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.736760 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:37:30.736848 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:37:30.736919 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":81,"threads_started":1}
I20260927 00:37:30.737171 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:30.731031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.10.11.194:40537)
0927 00:37:30.732197 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:30.732205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:30.732221 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:37:30.732275 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:30.732278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:30.732397 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:30.732558 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:30.732722 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.733907 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.735545 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:30.736541 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:30.736657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:30.736781 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:30.736930 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 00:37:30.737016 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":129,"threads_started":1}
I20260927 00:37:30.738586 27699 tablet_service.cc:1467] Processing CreateTablet for tablet 98ca67ab8e924e909d3831fd62c5021d (DEFAULT_TABLE table=test-table [id=77437e0f845c413095ee25d004b081b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:30.738586 27431 tablet_service.cc:1467] Processing CreateTablet for tablet 98ca67ab8e924e909d3831fd62c5021d (DEFAULT_TABLE table=test-table [id=77437e0f845c413095ee25d004b081b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:30.739610 27699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ca67ab8e924e909d3831fd62c5021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:30.739610 27431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ca67ab8e924e909d3831fd62c5021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:30.740233 27565 tablet_service.cc:1467] Processing CreateTablet for tablet 98ca67ab8e924e909d3831fd62c5021d (DEFAULT_TABLE table=test-table [id=77437e0f845c413095ee25d004b081b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:30.741115 27565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ca67ab8e924e909d3831fd62c5021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:30.747061 27782 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Bootstrap starting.
I20260927 00:37:30.747705 27783 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Bootstrap starting.
I20260927 00:37:30.747735 27784 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Bootstrap starting.
I20260927 00:37:30.749158 27782 tablet_bootstrap.cc:654] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:30.749766 27784 tablet_bootstrap.cc:654] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:30.749766 27783 tablet_bootstrap.cc:654] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:30.750336 27782 log.cc:824] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:30.751094 27783 log.cc:824] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:30.751094 27784 log.cc:824] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:30.752185 27782 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: No bootstrap required, opened a new log
I20260927 00:37:30.752372 27782 ts_tablet_manager.cc:1397] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:37:30.752823 27784 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: No bootstrap required, opened a new log
I20260927 00:37:30.752998 27784 ts_tablet_manager.cc:1397] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 00:37:30.753041 27783 tablet_bootstrap.cc:492] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: No bootstrap required, opened a new log
I20260927 00:37:30.753206 27783 ts_tablet_manager.cc:1397] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:37:30.756131 27784 raft_consensus.cc:348] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.756172 27782 raft_consensus.cc:348] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.756366 27784 raft_consensus.cc:374] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:30.756412 27782 raft_consensus.cc:374] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:30.756421 27784 raft_consensus.cc:729] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec6bc2f2be84e40b5ad4be3e20d05ab, State: Initialized, Role: FOLLOWER
I20260927 00:37:30.756464 27782 raft_consensus.cc:729] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcff2a6d59744f3abc8d5363fdd2663, State: Initialized, Role: FOLLOWER
I20260927 00:37:30.756855 27784 consensus_queue.cc:260] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.756872 27782 consensus_queue.cc:260] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.756901 27783 raft_consensus.cc:348] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.757133 27783 raft_consensus.cc:374] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:30.757200 27783 raft_consensus.cc:729] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d6f8d7fa4b4491a01feda14c727bc8, State: Initialized, Role: FOLLOWER
I20260927 00:37:30.757592 27631 heartbeater.cc:503] Master 127.10.11.254:43829 was elected leader, sending a full tablet report...
I20260927 00:37:30.757555 27783 consensus_queue.cc:260] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:30.757784 27784 ts_tablet_manager.cc:1428] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 00:37:30.757819 27782 ts_tablet_manager.cc:1428] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 00:37:30.758132 27497 heartbeater.cc:503] Master 127.10.11.254:43829 was elected leader, sending a full tablet report...
I20260927 00:37:30.758423 27783 ts_tablet_manager.cc:1428] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 00:37:30.758528 27765 heartbeater.cc:503] Master 127.10.11.254:43829 was elected leader, sending a full tablet report...
W20260927 00:37:30.776214 27498 tablet.cc:2369] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:37:30.938510 27766 tablet.cc:2369] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:37:30.978407 27632 tablet.cc:2369] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:31.059892 27788 raft_consensus.cc:482] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:31.060084 27788 raft_consensus.cc:504] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:31.061167 27788 leader_election.cc:288] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24d6f8d7fa4b4491a01feda14c727bc8 (127.10.11.195:40651), 9fcff2a6d59744f3abc8d5363fdd2663 (127.10.11.193:44599)
I20260927 00:37:31.066483 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.061531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:33159 (local address 127.10.11.193:44599)
0927 00:37:31.061659 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.061664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.061840 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:37:31.061976 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.061979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.062038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.062151 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:31.062534 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.063309 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065117 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.065749 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065878 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.065941 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.066262 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.066342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:37:31.066483 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.061509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:45535 (local address 127.10.11.195:40651)
0927 00:37:31.061642 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.061646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.061922 (+ 276us) server_negotiation.cc:407] Connection header received
0927 00:37:31.061990 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.061992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.062038 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.062150 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:31.062534 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.063312 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065117 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.065738 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065884 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.065938 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.066262 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.066342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:37:31.066545 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.061428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44599 (local address 127.10.11.194:33159)
0927 00:37:31.061721 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.061735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:31.061862 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:31.062163 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:31.062166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:31.062208 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:31.062414 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.062420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.063455 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.063458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.064986 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.064992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.065889 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.066075 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.066139 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 00:37:31.066228 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:37:31.066342 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":152,"spinlock_wait_cycles":4352,"thread_start_us":69,"threads_started":1}
I20260927 00:37:31.066542 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.061451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:40651 (local address 127.10.11.194:45535)
0927 00:37:31.061802 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.061813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:31.061918 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:31.062163 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:31.062166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:31.062206 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:31.062408 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.062415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.063455 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.063458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.064961 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.064973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.065889 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.065893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.066081 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.066143 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 00:37:31.066228 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:37:31.066345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":321,"spinlock_wait_cycles":28800,"thread_start_us":90,"threads_started":1}
I20260927 00:37:31.067226 27451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ca67ab8e924e909d3831fd62c5021d" candidate_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" is_pre_election: true
I20260927 00:37:31.067596 27451 raft_consensus.cc:2454] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec6bc2f2be84e40b5ad4be3e20d05ab in term 0.
I20260927 00:37:31.067534 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ca67ab8e924e909d3831fd62c5021d" candidate_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" is_pre_election: true
I20260927 00:37:31.067832 27719 raft_consensus.cc:2454] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec6bc2f2be84e40b5ad4be3e20d05ab in term 0.
I20260927 00:37:31.068146 27521 leader_election.cc:302] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec6bc2f2be84e40b5ad4be3e20d05ab, 9fcff2a6d59744f3abc8d5363fdd2663; no voters:
I20260927 00:37:31.068449 27788 raft_consensus.cc:2781] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:31.068531 27788 raft_consensus.cc:482] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:31.068573 27788 raft_consensus.cc:3037] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:31.069586 27788 raft_consensus.cc:504] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:31.069902 27788 leader_election.cc:288] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [CANDIDATE]: Term 1 election: Requested vote from peers 24d6f8d7fa4b4491a01feda14c727bc8 (127.10.11.195:40651), 9fcff2a6d59744f3abc8d5363fdd2663 (127.10.11.193:44599)
I20260927 00:37:31.070247 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ca67ab8e924e909d3831fd62c5021d" candidate_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d6f8d7fa4b4491a01feda14c727bc8"
I20260927 00:37:31.070261 27451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ca67ab8e924e909d3831fd62c5021d" candidate_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcff2a6d59744f3abc8d5363fdd2663"
I20260927 00:37:31.070382 27719 raft_consensus.cc:3037] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:31.070397 27451 raft_consensus.cc:3037] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:31.071373 27451 raft_consensus.cc:2454] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec6bc2f2be84e40b5ad4be3e20d05ab in term 1.
I20260927 00:37:31.071528 27719 raft_consensus.cc:2454] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec6bc2f2be84e40b5ad4be3e20d05ab in term 1.
I20260927 00:37:31.071681 27521 leader_election.cc:302] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec6bc2f2be84e40b5ad4be3e20d05ab, 9fcff2a6d59744f3abc8d5363fdd2663; no voters:
I20260927 00:37:31.071827 27788 raft_consensus.cc:2781] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:31.072041 27788 raft_consensus.cc:686] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 1 LEADER]: Becoming Leader. State: Replica: 3ec6bc2f2be84e40b5ad4be3e20d05ab, State: Running, Role: LEADER
I20260927 00:37:31.072345 27788 consensus_queue.cc:237] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } }
I20260927 00:37:31.075264 27305 catalog_manager.cc:5495] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ec6bc2f2be84e40b5ad4be3e20d05ab (127.10.11.194). New cstate: current_term: 1 leader_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec6bc2f2be84e40b5ad4be3e20d05ab" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 } health_report { overall_health: UNKNOWN } } }
I20260927 00:37:31.097445 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.093834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.10.11.194:40537)
0927 00:37:31.093950 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.093956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.094024 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:37:31.094223 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.094226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.094283 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.094383 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:31.094621 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.095108 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.095935 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.096165 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.096274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.096567 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:31.097071 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:31.097192 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.097261 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.097312 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:37:31.102494 27719 raft_consensus.cc:1261] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 1 FOLLOWER]: Refusing update from remote peer 3ec6bc2f2be84e40b5ad4be3e20d05ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:31.102524 27451 raft_consensus.cc:1261] T 98ca67ab8e924e909d3831fd62c5021d P 9fcff2a6d59744f3abc8d5363fdd2663 [term 1 FOLLOWER]: Refusing update from remote peer 3ec6bc2f2be84e40b5ad4be3e20d05ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:31.103139 27788 consensus_queue.cc:1035] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fcff2a6d59744f3abc8d5363fdd2663" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:31.103384 27793 consensus_queue.cc:1035] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "24d6f8d7fa4b4491a01feda14c727bc8" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:31.110268 27796 mvcc.cc:205] Tried to move back new op lower bound from 7333762871709814784 to 7333762871594598400. Current Snapshot: MvccSnapshot[applied={T|T < 7333762871709814784}]
I20260927 00:37:31.111867 27797 mvcc.cc:205] Tried to move back new op lower bound from 7333762871709814784 to 7333762871594598400. Current Snapshot: MvccSnapshot[applied={T|T < 7333762871709814784}]
I20260927 00:37:31.118209 27306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:31.118330 27306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:31.120512 27306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b0cc560dd04dc6ba68ea3e84132354: Sending DeleteTablet for 3 replicas of tablet 98ca67ab8e924e909d3831fd62c5021d
I20260927 00:37:31.121301 27699 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ca67ab8e924e909d3831fd62c5021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260927 00:37:31.121439 27565 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ca67ab8e924e909d3831fd62c5021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:37888
I20260927 00:37:31.121838 27807 tablet_replica.cc:330] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: stopping tablet replica
I20260927 00:37:31.121866 27431 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ca67ab8e924e909d3831fd62c5021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:31 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:57468
I20260927 00:37:31.122067 27806 tablet_replica.cc:330] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: stopping tablet replica
I20260927 00:37:31.122290 27807 raft_consensus.cc:2229] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:37:31.122308 27806 raft_consensus.cc:2229] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:31.122416 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27368
I20260927 00:37:31.122568 27806 raft_consensus.cc:2258] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:31.123901 27806 ts_tablet_manager.cc:1905] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:31.127148 27806 ts_tablet_manager.cc:1918] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:31.127249 27806 log.cc:1196] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/98ca67ab8e924e909d3831fd62c5021d
I20260927 00:37:31.127179 27807 raft_consensus.cc:2258] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:31.127607 27806 ts_tablet_manager.cc:1939] T 98ca67ab8e924e909d3831fd62c5021d P 24d6f8d7fa4b4491a01feda14c727bc8: Deleting consensus metadata
I20260927 00:37:31.128721 27807 ts_tablet_manager.cc:1905] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:31.128700 27292 catalog_manager.cc:4841] TS 24d6f8d7fa4b4491a01feda14c727bc8 (127.10.11.195:40651): tablet 98ca67ab8e924e909d3831fd62c5021d (table test-table [id=77437e0f845c413095ee25d004b081b6]) successfully deleted
I20260927 00:37:31.130786 27807 ts_tablet_manager.cc:1918] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:31.130899 27807 log.cc:1196] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/98ca67ab8e924e909d3831fd62c5021d
I20260927 00:37:31.131208 27807 ts_tablet_manager.cc:1939] T 98ca67ab8e924e909d3831fd62c5021d P 3ec6bc2f2be84e40b5ad4be3e20d05ab: Deleting consensus metadata
I20260927 00:37:31.132059 27292 catalog_manager.cc:4841] TS 3ec6bc2f2be84e40b5ad4be3e20d05ab (127.10.11.194:40537): tablet 98ca67ab8e924e909d3831fd62c5021d (table test-table [id=77437e0f845c413095ee25d004b081b6]) successfully deleted
W20260927 00:37:31.132495 27293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:44599 (error 108)
I20260927 00:37:31.132689 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27502
I20260927 00:37:31.134358 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.134016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44599 (local address 127.0.0.1:57476)
0927 00:37:31.134167 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.134261 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:37:31.134509 27293 catalog_manager.cc:4584] TS 9fcff2a6d59744f3abc8d5363fdd2663 (127.10.11.193:44599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98ca67ab8e924e909d3831fd62c5021d: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44599: connect: Connection refused (error 111)
I20260927 00:37:31.138468 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27636
I20260927 00:37:31.143901 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27274
2026-09-27T00:37:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2471 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:31Z Disabled control of system clock
I20260927 00:37:31.173619 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45551
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:45551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:31.285190 27813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:31.285422 27813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:31.285462 27813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:31.289942 27813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:31.290000 27813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:31.290024 27813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:31.290041 27813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:31.290056 27813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:31.295425 27813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:45551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:31.296530 27813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:31.298014 27813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:31.306833 27819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.306903 27818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.306954 27813 server_base.cc:1034] running on GCE node
W20260927 00:37:31.307072 27821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.307715 27813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:31.308732 27813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:31.309916 27813 hybrid_clock.cc:648] HybridClock initialized: now 1790469451309898 us; error 28 us; skew 500 ppm
I20260927 00:37:31.311666 27813 webserver.cc:460] Webserver started at http://127.10.11.254:43405/ using document root <none> and password file <none>
I20260927 00:37:31.312207 27813 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:31.312294 27813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:31.312490 27813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:31.314463 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "68bd3942fc0743ccd2755ec5e56ee21e"
server_key_iv: "d733eb9145616c646f734df7d68a3aa5"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.315028 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "68bd3942fc0743ccd2755ec5e56ee21e"
server_key_iv: "d733eb9145616c646f734df7d68a3aa5"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.318398 27813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:31.320633 27827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.321736 27813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:31.321867 27813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "68bd3942fc0743ccd2755ec5e56ee21e"
server_key_iv: "d733eb9145616c646f734df7d68a3aa5"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.321969 27813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:31.333155 27813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:31.333760 27813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:31.333943 27813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:31.340370 27813 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:45551
I20260927 00:37:31.340376 27879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:45551 every 8 connection(s)
I20260927 00:37:31.341218 27813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:31.344080 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:31.349053 27880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: Bootstrap starting.
I20260927 00:37:31.349932 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27813
I20260927 00:37:31.350112 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:31.350411 10287 external_mini_cluster.cc:1428] Setting key 42971368d62d69e6f85f74efcf44c834
I20260927 00:37:31.351794 27880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:31.352478 27880 log.cc:824] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:31.354264 27880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: No bootstrap required, opened a new log
I20260927 00:37:31.357585 27880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } }
I20260927 00:37:31.357810 27880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:31.357870 27880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346dc02f173f44f2b7f2c9f6b5eb8f2c, State: Initialized, Role: FOLLOWER
I20260927 00:37:31.358367 27880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } }
I20260927 00:37:31.358487 27880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:31.358590 27880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:31.358711 27880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:31.359230 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.351403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.10.11.254:45551)
0927 00:37:31.352009 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.352029 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.352062 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:31.352687 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.352705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.353072 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.353384 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:31.353781 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.354828 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.355720 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.356192 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.356377 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.357269 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:31.357274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:31.357286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:31.357635 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:31.357685 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:31.357705 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:31.357814 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.358039 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.358310 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260927 00:37:31.359601 27880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } }
I20260927 00:37:31.359917 27880 leader_election.cc:302] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 346dc02f173f44f2b7f2c9f6b5eb8f2c; no voters:
I20260927 00:37:31.360226 27880 leader_election.cc:288] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:31.360352 27885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:31.360584 27885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [term 1 LEADER]: Becoming Leader. State: Replica: 346dc02f173f44f2b7f2c9f6b5eb8f2c, State: Running, Role: LEADER
I20260927 00:37:31.360863 27885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } }
I20260927 00:37:31.361382 27880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:31.361912 27885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346dc02f173f44f2b7f2c9f6b5eb8f2c. Latest consensus state: current_term: 1 leader_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } } }
I20260927 00:37:31.362049 27885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:31.362984 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346dc02f173f44f2b7f2c9f6b5eb8f2c" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 45551 } } }
I20260927 00:37:31.363087 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:31.363626 27892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:31.366195 27892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:31.370916 27892 catalog_manager.cc:1347] Generated new cluster ID: 5c8f596c7ddf4468a0ad3c6ba40d0944
I20260927 00:37:31.370978 27892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:31.401810 27892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:31.402729 27892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:31.410575 27892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: Generated new TSK 0
I20260927 00:37:31.411231 27892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:31.419634 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--builtin_ntp_servers=127.10.11.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:31.525732 27904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:31.525957 27904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:31.525993 27904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:31.529903 27904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:31.529948 27904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:31.530030 27904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:31.534572 27904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:31.535496 27904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:31.536746 27904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:31.545747 27909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.545809 27912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.545781 27910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.546042 27904 server_base.cc:1034] running on GCE node
I20260927 00:37:31.546412 27904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:31.546993 27904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:31.548178 27904 hybrid_clock.cc:648] HybridClock initialized: now 1790469451548169 us; error 31 us; skew 500 ppm
I20260927 00:37:31.549849 27904 webserver.cc:460] Webserver started at http://127.10.11.193:43179/ using document root <none> and password file <none>
I20260927 00:37:31.550467 27904 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:31.550558 27904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:31.550746 27904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:31.552675 27904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "ae57e54628f6405db505a656f18f9048"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "aae5a86567ad62bed342ee030e141948"
server_key_iv: "67e7e6cc57a594283943b9b8113fbe70"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.553212 27904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "ae57e54628f6405db505a656f18f9048"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "aae5a86567ad62bed342ee030e141948"
server_key_iv: "67e7e6cc57a594283943b9b8113fbe70"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.556538 27904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:37:31.558709 27918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.559893 27904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:31.560006 27904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "ae57e54628f6405db505a656f18f9048"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "aae5a86567ad62bed342ee030e141948"
server_key_iv: "67e7e6cc57a594283943b9b8113fbe70"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.560094 27904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:31.584515 27904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:31.585041 27904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:31.585222 27904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:31.585738 27904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:31.586644 27904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:31.586690 27904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.586752 27904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:31.586783 27904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.595513 27904 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:39521
I20260927 00:37:31.595561 28031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:39521 every 8 connection(s)
I20260927 00:37:31.596349 27904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:31.604722 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.598065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:60297 (local address 127.10.11.254:45551)
0927 00:37:31.598225 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.598230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.599192 (+ 962us) server_negotiation.cc:407] Connection header received
0927 00:37:31.600530 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.600533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.600574 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.600645 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:31.601442 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.601966 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.603058 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.603263 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.603338 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.604356 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:31.604358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:31.604360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:31.604370 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:31.604384 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:31.604390 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:31.604442 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.604576 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.604617 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:37:31.605515 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.598442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45551 (local address 127.10.11.193:60297)
0927 00:37:31.599062 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.599098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:31.600311 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:31.600822 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:31.600828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:31.600913 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:31.601331 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.601341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.602083 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.602086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.602950 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.602959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.603386 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.603389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.603535 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.604141 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:31.604142 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:31.604178 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:31.604233 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:31.604461 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:31.604467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:31.604471 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:37:31.604561 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:37:31.604751 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260927 00:37:31.605907 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 27904
I20260927 00:37:31.606057 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:31.606405 10287 external_mini_cluster.cc:1428] Setting key 80cf824f4d874894f968c429243e3362
I20260927 00:37:31.606810 28032 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45551
I20260927 00:37:31.607043 28032 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:31.607549 28032 heartbeater.cc:511] Master 127.10.11.254:45551 requested a full tablet report, sending...
I20260927 00:37:31.608233 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--builtin_ntp_servers=127.10.11.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:31.609118 27844 ts_manager.cc:194] Registered new tserver with Master: ae57e54628f6405db505a656f18f9048 (127.10.11.193:39521)
I20260927 00:37:31.610339 27844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.193:60297
W20260927 00:37:31.712750 28035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:31.712975 28035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:31.713008 28035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:31.716867 28035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:31.716910 28035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:31.716961 28035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:31.721314 28035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:31.722126 28035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:31.723250 28035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:31.732411 28041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.732420 28040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.732452 28043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.732686 28035 server_base.cc:1034] running on GCE node
I20260927 00:37:31.733036 28035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:31.733604 28035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:31.734786 28035 hybrid_clock.cc:648] HybridClock initialized: now 1790469451734759 us; error 40 us; skew 500 ppm
I20260927 00:37:31.736465 28035 webserver.cc:460] Webserver started at http://127.10.11.194:45517/ using document root <none> and password file <none>
I20260927 00:37:31.736989 28035 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:31.737076 28035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:31.737283 28035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:31.739384 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "39dd92206e65481b9ab38806a2b1580c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "556e28f0159e5369772a0760aa29a3a1"
server_key_iv: "fb46cb1bed5d0df048053a426523b61d"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.739907 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "39dd92206e65481b9ab38806a2b1580c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "556e28f0159e5369772a0760aa29a3a1"
server_key_iv: "fb46cb1bed5d0df048053a426523b61d"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.743314 28035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:31.745440 28049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.746585 28035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:31.746707 28035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "39dd92206e65481b9ab38806a2b1580c"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "556e28f0159e5369772a0760aa29a3a1"
server_key_iv: "fb46cb1bed5d0df048053a426523b61d"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.746807 28035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:31.783569 28035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:31.784271 28035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:31.784466 28035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:31.785017 28035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:31.785912 28035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:31.785985 28035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.786055 28035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:31.786098 28035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.794592 28035 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:39409
I20260927 00:37:31.794633 28162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:39409 every 8 connection(s)
I20260927 00:37:31.795418 28035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:31.803791 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.797081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:35319 (local address 127.10.11.254:45551)
0927 00:37:31.797209 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.797213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.798301 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 00:37:31.799679 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.799681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.799724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.799796 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:31.800591 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.801110 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.802174 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.802397 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.802472 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.803408 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:31.803411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:31.803413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:31.803423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:31.803440 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:31.803447 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:31.803506 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.803638 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.803681 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:37:31.804579 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.797449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45551 (local address 127.10.11.194:35319)
0927 00:37:31.798047 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.798083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:31.799497 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:31.799952 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:31.799958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:31.800033 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:31.800473 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.800484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.801209 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.801211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.802046 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.802054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.802515 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.802518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.802642 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.803158 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:31.803159 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:31.803194 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:31.803249 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:31.803519 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:31.803525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:31.803529 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:37:31.803615 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:37:31.803807 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260927 00:37:31.804733 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28035
I20260927 00:37:31.804848 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:37:31.805169 10287 external_mini_cluster.cc:1428] Setting key 7f4402da3fb479435d002d4a8003898b
I20260927 00:37:31.805749 28163 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45551
I20260927 00:37:31.805974 28163 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:31.806499 28163 heartbeater.cc:511] Master 127.10.11.254:45551 requested a full tablet report, sending...
I20260927 00:37:31.807240 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--builtin_ntp_servers=127.10.11.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:31.807499 27844 ts_manager.cc:194] Registered new tserver with Master: 39dd92206e65481b9ab38806a2b1580c (127.10.11.194:39409)
I20260927 00:37:31.808307 27844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.194:35319
W20260927 00:37:31.914767 28166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:31.915006 28166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:31.915043 28166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:31.918970 28166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:31.919010 28166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:31.919090 28166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:31.923619 28166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:31.924503 28166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:31.925794 28166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:31.934988 28171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:31.935024 28172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.935053 28166 server_base.cc:1034] running on GCE node
W20260927 00:37:31.934988 28174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:31.935597 28166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:31.936182 28166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:31.937378 28166 hybrid_clock.cc:648] HybridClock initialized: now 1790469451937356 us; error 37 us; skew 500 ppm
I20260927 00:37:31.939121 28166 webserver.cc:460] Webserver started at http://127.10.11.195:43235/ using document root <none> and password file <none>
I20260927 00:37:31.939659 28166 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:31.939747 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:31.939963 28166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:31.941835 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "133e40778dc3ae5ff0fed955d321667e"
server_key_iv: "40d4d79e913fa2e8089fc3a3c089af09"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.942385 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "133e40778dc3ae5ff0fed955d321667e"
server_key_iv: "40d4d79e913fa2e8089fc3a3c089af09"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.945678 28166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:37:31.948173 28180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.949292 28166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:37:31.949425 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0"
format_stamp: "Formatted at 2026-09-27 00:37:31 on dist-test-slave-6gxc"
server_key: "133e40778dc3ae5ff0fed955d321667e"
server_key_iv: "40d4d79e913fa2e8089fc3a3c089af09"
server_key_version: "encryptionkey@0"
I20260927 00:37:31.949533 28166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:31.967185 28166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:31.967787 28166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:31.967971 28166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:31.968518 28166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:31.969394 28166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:31.969465 28166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.969532 28166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:31.969590 28166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:31.978310 28166 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:33675
I20260927 00:37:31.978343 28293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:33675 every 8 connection(s)
I20260927 00:37:31.979139 28166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:31.983237 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28166
I20260927 00:37:31.983362 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:31.983695 10287 external_mini_cluster.cc:1428] Setting key 39146a5da7e98475dad4f37ff90b4c54
I20260927 00:37:31.988546 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.980791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:39635 (local address 127.10.11.254:45551)
0927 00:37:31.980939 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.980943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.981878 (+ 935us) server_negotiation.cc:407] Connection header received
0927 00:37:31.983392 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.983397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.983461 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.983542 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:31.984667 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.985428 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.986766 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.987010 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.987101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.988166 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:31.988169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:31.988171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:31.988181 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:31.988196 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:31.988202 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:31.988255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:31.988396 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:37:31.988441 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:37:31.989346 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.981132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:45551 (local address 127.10.11.195:39635)
0927 00:37:31.981735 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:31.981772 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:31.983129 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:31.983749 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:31.983757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:31.983868 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:31.984490 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.984510 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.985549 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.985551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.986651 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:31.986660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:31.987143 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:31.987145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:31.987304 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:31.987925 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:31.987926 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:31.987965 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:31.988025 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:31.988274 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:31.988280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:31.988284 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:37:31.988375 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:37:31.988581 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260927 00:37:31.990401 28294 heartbeater.cc:347] Connected to a master server at 127.10.11.254:45551
I20260927 00:37:31.990597 28294 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:31.991026 28294 heartbeater.cc:511] Master 127.10.11.254:45551 requested a full tablet report, sending...
I20260927 00:37:31.991923 27844 ts_manager.cc:194] Registered new tserver with Master: c7a82c6a6d34490ba5bc1a00ef20e6d0 (127.10.11.195:33675)
I20260927 00:37:31.992466 27844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.195:39635
I20260927 00:37:31.997490 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:32.003203 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:31.999396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.10.11.254:45551)
0927 00:37:31.999544 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:31.999550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:31.999712 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:37:31.999852 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:31.999855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:31.999895 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:31.999992 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:32.000256 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.000760 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.001591 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.001819 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.001929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.002798 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:32.002802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:32.002804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:32.002816 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:32.002835 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:32.002848 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:32.002940 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.003027 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.003075 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:37:32.006052 27844 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:32.009042 27844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:32.022470 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.016829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.10.11.194:39409)
0927 00:37:32.017265 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.017279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.017305 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:37:32.017380 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.017385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.017594 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.017770 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:32.018289 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.019465 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021151 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.021727 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.022042 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.022168 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.022268 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20260927 00:37:32.022655 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.017018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39521 (local address 127.0.0.1:41838)
0927 00:37:32.017513 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.017528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:32.017612 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:32.018603 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:32.018606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:32.018623 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:32.018720 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.018730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.020038 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.020041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.021170 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.021177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.022166 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.022170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.022313 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.022342 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:37:32.022413 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:37:32.022514 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260927 00:37:32.022719 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.017419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.10.11.193:39521)
0927 00:37:32.018167 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.018181 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.018204 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:37:32.018279 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.018284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.018468 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.018613 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:32.018846 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.019931 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021287 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.022001 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.022257 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.022442 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.022492 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.022573 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":95,"threads_started":1}
I20260927 00:37:32.022770 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.017741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.10.11.195:33675)
0927 00:37:32.018181 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.018194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.018217 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:37:32.018289 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.018294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.018491 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.018635 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:32.018971 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.019801 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021102 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.021852 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021965 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.022187 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.022502 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.022611 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":127,"threads_started":1}
I20260927 00:37:32.023196 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.017354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33675 (local address 127.0.0.1:47454)
0927 00:37:32.017795 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.017807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:32.017888 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:32.018758 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:32.018764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:32.018784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:32.018866 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.018870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.019932 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.019936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.020970 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.020979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.022152 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.022155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.022260 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.022280 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:37:32.022347 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:37:32.023037 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":557,"thread_start_us":93,"threads_started":1}
I20260927 00:37:32.023221 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.016641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:39409 (local address 127.0.0.1:46314)
0927 00:37:32.017017 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.017067 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:32.017189 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:32.017965 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:32.017970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:32.017997 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:32.018154 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.018162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.019593 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.019596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.021014 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.021025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.021850 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.021854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.021954 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.021988 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:37:32.022972 (+ 984us) client_negotiation.cc:240] Negotiation successful
0927 00:37:32.023065 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":60,"thread_start_us":106,"threads_started":1}
I20260927 00:37:32.025763 27966 tablet_service.cc:1467] Processing CreateTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 (DEFAULT_TABLE table=test-table [id=73d4606ca58f4f7494f15b97f29ba4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:32.025763 28097 tablet_service.cc:1467] Processing CreateTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 (DEFAULT_TABLE table=test-table [id=73d4606ca58f4f7494f15b97f29ba4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:32.025760 28228 tablet_service.cc:1467] Processing CreateTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 (DEFAULT_TABLE table=test-table [id=73d4606ca58f4f7494f15b97f29ba4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:32.026741 28228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b95cec67f846b1b365e1e9d51af2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:32.026765 27966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b95cec67f846b1b365e1e9d51af2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:32.026765 28097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b95cec67f846b1b365e1e9d51af2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:32.032948 28311 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Bootstrap starting.
I20260927 00:37:32.033914 28313 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Bootstrap starting.
I20260927 00:37:32.033910 28312 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Bootstrap starting.
I20260927 00:37:32.035110 28311 tablet_bootstrap.cc:654] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:32.035808 28311 log.cc:824] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:32.035952 28312 tablet_bootstrap.cc:654] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:32.035953 28313 tablet_bootstrap.cc:654] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:32.036767 28313 log.cc:824] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:32.036767 28312 log.cc:824] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:32.038079 28311 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: No bootstrap required, opened a new log
I20260927 00:37:32.038301 28311 ts_tablet_manager.cc:1397] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260927 00:37:32.038666 28313 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: No bootstrap required, opened a new log
I20260927 00:37:32.038859 28313 ts_tablet_manager.cc:1397] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:37:32.039029 28312 tablet_bootstrap.cc:492] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: No bootstrap required, opened a new log
I20260927 00:37:32.039201 28312 ts_tablet_manager.cc:1397] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:37:32.041805 28313 raft_consensus.cc:348] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.042040 28313 raft_consensus.cc:374] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:32.042100 28313 raft_consensus.cc:729] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39dd92206e65481b9ab38806a2b1580c, State: Initialized, Role: FOLLOWER
I20260927 00:37:32.042533 28313 consensus_queue.cc:260] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.042546 28311 raft_consensus.cc:348] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.042788 28311 raft_consensus.cc:374] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:32.042850 28311 raft_consensus.cc:729] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a82c6a6d34490ba5bc1a00ef20e6d0, State: Initialized, Role: FOLLOWER
I20260927 00:37:32.043175 28163 heartbeater.cc:503] Master 127.10.11.254:45551 was elected leader, sending a full tablet report...
I20260927 00:37:32.043157 28312 raft_consensus.cc:348] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.043275 28311 consensus_queue.cc:260] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.043366 28312 raft_consensus.cc:374] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:32.043409 28312 raft_consensus.cc:729] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae57e54628f6405db505a656f18f9048, State: Initialized, Role: FOLLOWER
I20260927 00:37:32.043467 28313 ts_tablet_manager.cc:1428] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:37:32.043797 28312 consensus_queue.cc:260] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.044186 28311 ts_tablet_manager.cc:1428] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 00:37:32.044495 28294 heartbeater.cc:503] Master 127.10.11.254:45551 was elected leader, sending a full tablet report...
I20260927 00:37:32.044561 28032 heartbeater.cc:503] Master 127.10.11.254:45551 was elected leader, sending a full tablet report...
I20260927 00:37:32.044652 28312 ts_tablet_manager.cc:1428] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 00:37:32.046227 28164 tablet.cc:2369] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:32.069916 28317 raft_consensus.cc:482] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:32.070088 28317 raft_consensus.cc:504] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.071194 28317 leader_election.cc:288] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7a82c6a6d34490ba5bc1a00ef20e6d0 (127.10.11.195:33675), ae57e54628f6405db505a656f18f9048 (127.10.11.193:39521)
I20260927 00:37:32.074932 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.071375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:33675 (local address 127.10.11.194:34187)
0927 00:37:32.071536 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.071552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:32.071637 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:32.071966 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:32.071969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:32.071986 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:32.072133 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.072142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.072876 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.072881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.073706 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.073714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.074518 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.074524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.074616 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.074644 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:37:32.074712 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:37:32.074796 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":32}
I20260927 00:37:32.074947 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.071573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:34187 (local address 127.10.11.195:33675)
0927 00:37:32.071722 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.071729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.071746 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:37:32.071800 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.071803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.071853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.071941 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:32.072257 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.072760 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.073831 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.074391 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.074522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.074572 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.074748 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.074820 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:37:32.074963 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.071409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39521 (local address 127.10.11.194:37149)
0927 00:37:32.071703 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.071718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:32.071811 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:32.072082 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:32.072084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:32.072097 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:32.072188 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.072216 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.072944 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.072946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.073798 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:32.073809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.074563 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.074566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:32.074661 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.074678 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:37:32.074772 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:37:32.074831 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":37,"thread_start_us":101,"threads_started":1}
I20260927 00:37:32.074965 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.071504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:37149 (local address 127.10.11.193:39521)
0927 00:37:32.071740 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.071745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.071784 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:37:32.071929 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.071934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.071984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.072059 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:32.072328 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.072834 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.073925 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.074432 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.074562 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.074618 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.074797 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.074861 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260927 00:37:32.075562 27986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b95cec67f846b1b365e1e9d51af2a4" candidate_uuid: "39dd92206e65481b9ab38806a2b1580c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae57e54628f6405db505a656f18f9048" is_pre_election: true
I20260927 00:37:32.075760 28248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b95cec67f846b1b365e1e9d51af2a4" candidate_uuid: "39dd92206e65481b9ab38806a2b1580c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" is_pre_election: true
I20260927 00:37:32.075846 27986 raft_consensus.cc:2454] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39dd92206e65481b9ab38806a2b1580c in term 0.
I20260927 00:37:32.076033 28248 raft_consensus.cc:2454] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39dd92206e65481b9ab38806a2b1580c in term 0.
I20260927 00:37:32.076325 28050 leader_election.cc:302] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39dd92206e65481b9ab38806a2b1580c, ae57e54628f6405db505a656f18f9048; no voters:
I20260927 00:37:32.076577 28317 raft_consensus.cc:2781] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:32.076653 28317 raft_consensus.cc:482] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:32.076700 28317 raft_consensus.cc:3037] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:32.077922 28317 raft_consensus.cc:504] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.078293 28317 leader_election.cc:288] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [CANDIDATE]: Term 1 election: Requested vote from peers c7a82c6a6d34490ba5bc1a00ef20e6d0 (127.10.11.195:33675), ae57e54628f6405db505a656f18f9048 (127.10.11.193:39521)
I20260927 00:37:32.078578 28248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b95cec67f846b1b365e1e9d51af2a4" candidate_uuid: "39dd92206e65481b9ab38806a2b1580c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0"
I20260927 00:37:32.078601 27986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b95cec67f846b1b365e1e9d51af2a4" candidate_uuid: "39dd92206e65481b9ab38806a2b1580c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae57e54628f6405db505a656f18f9048"
I20260927 00:37:32.078701 27986 raft_consensus.cc:3037] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:32.078715 28248 raft_consensus.cc:3037] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:32.079591 27986 raft_consensus.cc:2454] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39dd92206e65481b9ab38806a2b1580c in term 1.
I20260927 00:37:32.079735 28248 raft_consensus.cc:2454] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39dd92206e65481b9ab38806a2b1580c in term 1.
I20260927 00:37:32.079849 28050 leader_election.cc:302] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39dd92206e65481b9ab38806a2b1580c, ae57e54628f6405db505a656f18f9048; no voters:
I20260927 00:37:32.080036 28317 raft_consensus.cc:2781] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:32.080250 28317 raft_consensus.cc:686] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 1 LEADER]: Becoming Leader. State: Replica: 39dd92206e65481b9ab38806a2b1580c, State: Running, Role: LEADER
I20260927 00:37:32.080562 28317 consensus_queue.cc:237] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } }
I20260927 00:37:32.083499 27844 catalog_manager.cc:5495] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c reported cstate change: term changed from 0 to 1, leader changed from <none> to 39dd92206e65481b9ab38806a2b1580c (127.10.11.194). New cstate: current_term: 1 leader_uuid: "39dd92206e65481b9ab38806a2b1580c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39dd92206e65481b9ab38806a2b1580c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 39409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 } health_report { overall_health: UNKNOWN } } }
W20260927 00:37:32.097414 28033 tablet.cc:2369] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:32.100531 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.097034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.10.11.194:39409)
0927 00:37:32.097144 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:32.097147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:32.097159 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:37:32.097262 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:32.097264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:32.097309 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:32.097393 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:32.097676 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.098220 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.099166 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:32.099399 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:32.099501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:32.099791 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:32.100145 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:32.100265 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:32.100339 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:37:32.100385 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:37:32.105517 27986 raft_consensus.cc:1261] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 1 FOLLOWER]: Refusing update from remote peer 39dd92206e65481b9ab38806a2b1580c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:32.105518 28248 raft_consensus.cc:1261] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 1 FOLLOWER]: Refusing update from remote peer 39dd92206e65481b9ab38806a2b1580c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:32.106190 28321 consensus_queue.cc:1035] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae57e54628f6405db505a656f18f9048" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:32.106379 28317 consensus_queue.cc:1035] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a82c6a6d34490ba5bc1a00ef20e6d0" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:32.111999 28324 mvcc.cc:205] Tried to move back new op lower bound from 7333762875818258432 to 7333762875724214272. Current Snapshot: MvccSnapshot[applied={T|T < 7333762875818258432}]
I20260927 00:37:32.119765 27844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:32.119927 27844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:32.122216 27844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 346dc02f173f44f2b7f2c9f6b5eb8f2c: Sending DeleteTablet for 3 replicas of tablet 92b95cec67f846b1b365e1e9d51af2a4
I20260927 00:37:32.123011 28228 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:32 UTC) from {username='slave'} at 127.0.0.1:47454
I20260927 00:37:32.123009 28097 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:32 UTC) from {username='slave'} at 127.0.0.1:46314
I20260927 00:37:32.123051 27966 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b95cec67f846b1b365e1e9d51af2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:32 UTC) from {username='slave'} at 127.0.0.1:41838
I20260927 00:37:32.123561 28336 tablet_replica.cc:330] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048: stopping tablet replica
I20260927 00:37:32.123564 28335 tablet_replica.cc:330] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: stopping tablet replica
I20260927 00:37:32.123843 28335 raft_consensus.cc:2229] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:37:32.123843 28336 raft_consensus.cc:2229] T 92b95cec67f846b1b365e1e9d51af2a4 P ae57e54628f6405db505a656f18f9048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:32.124107 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27904
I20260927 00:37:32.124152 28335 raft_consensus.cc:2258] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:32.125850 28334 tablet_replica.cc:330] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: stopping tablet replica
I20260927 00:37:32.126194 28334 raft_consensus.cc:2229] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:32.126459 28334 raft_consensus.cc:2258] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:32.128000 28334 ts_tablet_manager.cc:1905] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:32.128014 28335 ts_tablet_manager.cc:1905] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:32.130707 28334 ts_tablet_manager.cc:1918] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:32.130802 28334 log.cc:1196] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/92b95cec67f846b1b365e1e9d51af2a4
I20260927 00:37:32.131114 28334 ts_tablet_manager.cc:1939] T 92b95cec67f846b1b365e1e9d51af2a4 P c7a82c6a6d34490ba5bc1a00ef20e6d0: Deleting consensus metadata
I20260927 00:37:32.131319 28335 ts_tablet_manager.cc:1918] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:32.131558 28335 log.cc:1196] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/92b95cec67f846b1b365e1e9d51af2a4
I20260927 00:37:32.131894 28335 ts_tablet_manager.cc:1939] T 92b95cec67f846b1b365e1e9d51af2a4 P 39dd92206e65481b9ab38806a2b1580c: Deleting consensus metadata
I20260927 00:37:32.132265 27830 catalog_manager.cc:4841] TS c7a82c6a6d34490ba5bc1a00ef20e6d0 (127.10.11.195:33675): tablet 92b95cec67f846b1b365e1e9d51af2a4 (table test-table [id=73d4606ca58f4f7494f15b97f29ba4a3]) successfully deleted
I20260927 00:37:32.132932 27830 catalog_manager.cc:4841] TS 39dd92206e65481b9ab38806a2b1580c (127.10.11.194:39409): tablet 92b95cec67f846b1b365e1e9d51af2a4 (table test-table [id=73d4606ca58f4f7494f15b97f29ba4a3]) successfully deleted
W20260927 00:37:32.133244 27828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:39521 (error 108)
I20260927 00:37:32.133920 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28035
I20260927 00:37:32.134238 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:32.133941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:39521 (local address 127.0.0.1:41854)
0927 00:37:32.134064 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:32.134152 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:39521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:37:32.134378 27828 catalog_manager.cc:4584] TS ae57e54628f6405db505a656f18f9048 (127.10.11.193:39521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92b95cec67f846b1b365e1e9d51af2a4: Network error: Client connection negotiation failed: client connection to 127.10.11.193:39521: connect: Connection refused (error 111)
I20260927 00:37:32.139994 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28166
I20260927 00:37:32.145359 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 27813
2026-09-27T00:37:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1000 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:32 dist-test-slave-6gxc krb5kdc[28340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:32 dist-test-slave-6gxc krb5kdc[28340](info): set up 2 sockets
Sep 27 00:37:32 dist-test-slave-6gxc krb5kdc[28340](info): commencing operation
krb5kdc: starting...
W20260927 00:37:33.193858 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:33.307031 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42061
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:33.411413 28353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:33.411621 28353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:33.411675 28353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:33.415642 28353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:33.415688 28353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:33.415707 28353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:33.415723 28353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:33.415738 28353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:33.420626 28353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:33.421607 28353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:33.422556 28353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:33.430871 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:33.430863 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:33.430886 28361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:33.431206 28353 server_base.cc:1034] running on GCE node
I20260927 00:37:33.431730 28353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:33.432662 28353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:33.433822 28353 hybrid_clock.cc:648] HybridClock initialized: now 1790469453433794 us; error 40 us; skew 500 ppm
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:33.436498 28353 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:33.437287 28353 webserver.cc:460] Webserver started at http://127.10.11.254:33667/ using document root <none> and password file <none>
I20260927 00:37:33.437749 28353 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:33.437803 28353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:33.437944 28353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:33.439896 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "ef19aedfe57841fa8c4528378b7eaf95"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2649d7e33e840b3ee06b249faf509ea6"
server_key_iv: "c66971a1622800e54817b5383873580f"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.440377 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "ef19aedfe57841fa8c4528378b7eaf95"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2649d7e33e840b3ee06b249faf509ea6"
server_key_iv: "c66971a1622800e54817b5383873580f"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.443746 28353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:37:33.445883 28368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.447057 28353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:37:33.447192 28353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "ef19aedfe57841fa8c4528378b7eaf95"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2649d7e33e840b3ee06b249faf509ea6"
server_key_iv: "c66971a1622800e54817b5383873580f"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.447295 28353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:33.470744 28353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:33.473749 28353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:33.473944 28353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:33.480813 28353 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42061
I20260927 00:37:33.480839 28420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42061 every 8 connection(s)
I20260927 00:37:33.481657 28353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:33.482033 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28353
I20260927 00:37:33.482208 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:33.482472 10287 external_mini_cluster.cc:1428] Setting key 0c63fdc914ae2114ca410eb5857ab48c
I20260927 00:37:33.484483 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:33.490736 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: Bootstrap starting.
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:33.493019 28421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:33.493901 28421 log.cc:824] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:33.496064 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: No bootstrap required, opened a new log
I20260927 00:37:33.497124 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:33.483732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.10.11.254:42061)
0927 00:37:33.484127 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:33.484144 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:33.484185 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:37:33.484984 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:33.485006 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:33.485395 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:33.485723 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:33.486420 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.487899 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.489016 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.489697 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.489853 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:33.492269 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:33.492275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:33.492290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:33.492319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:33.494243 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.494782 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.494790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.494796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.494921 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.495170 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.495173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.495175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.495624 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:33.495809 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:33.495986 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:37:33.496267 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260927 00:37:33.499882 28421 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } }
I20260927 00:37:33.500073 28421 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:33.500108 28421 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef19aedfe57841fa8c4528378b7eaf95, State: Initialized, Role: FOLLOWER
I20260927 00:37:33.500490 28421 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } }
I20260927 00:37:33.500612 28421 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:33.500656 28421 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:33.500717 28421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:33.501451 28421 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } }
I20260927 00:37:33.501813 28421 leader_election.cc:302] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef19aedfe57841fa8c4528378b7eaf95; no voters:
I20260927 00:37:33.502175 28421 leader_election.cc:288] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:33.502259 28426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:33.502466 28426 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [term 1 LEADER]: Becoming Leader. State: Replica: ef19aedfe57841fa8c4528378b7eaf95, State: Running, Role: LEADER
I20260927 00:37:33.502753 28426 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } }
I20260927 00:37:33.503325 28421 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:33.504298 28427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef19aedfe57841fa8c4528378b7eaf95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } } }
I20260927 00:37:33.504442 28427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:33.504793 28435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:33.504796 28428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef19aedfe57841fa8c4528378b7eaf95. Latest consensus state: current_term: 1 leader_uuid: "ef19aedfe57841fa8c4528378b7eaf95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef19aedfe57841fa8c4528378b7eaf95" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42061 } } }
I20260927 00:37:33.505466 28428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:33.507797 28435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:33.512884 28435 catalog_manager.cc:1347] Generated new cluster ID: 21b6eba226db4f16bd30f36459cab9e7
I20260927 00:37:33.512953 28435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:33.528788 28435 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:33.529587 28435 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:33.538501 28435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: Generated new TSK 0
I20260927 00:37:33.539069 28435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:37:33.574312 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:33.690874 28447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:33.691087 28447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:33.691126 28447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:33.694991 28447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:33.695034 28447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:33.695124 28447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:33.700001 28447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:33.700863 28447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:33.701761 28447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:33.710807 28455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:33.710865 28447 server_base.cc:1034] running on GCE node
W20260927 00:37:33.710876 28452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:33.710814 28453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:33.711340 28447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:33.711907 28447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:33.713095 28447 hybrid_clock.cc:648] HybridClock initialized: now 1790469453713067 us; error 40 us; skew 500 ppm
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:33.715803 28447 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:37:33.716626 28447 webserver.cc:460] Webserver started at http://127.10.11.193:33829/ using document root <none> and password file <none>
I20260927 00:37:33.717135 28447 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:33.717223 28447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:33.717444 28447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:33.719374 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "9ce13fb1fa5244238cec886a0d2260c3"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "236314ddb13809975fc8929a736a1fa5"
server_key_iv: "96e6d6377b561f8d8f40f4e07916c9d9"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.719908 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "9ce13fb1fa5244238cec886a0d2260c3"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "236314ddb13809975fc8929a736a1fa5"
server_key_iv: "96e6d6377b561f8d8f40f4e07916c9d9"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.723284 28447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:33.725493 28462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.726677 28447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:33.726771 28447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "9ce13fb1fa5244238cec886a0d2260c3"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "236314ddb13809975fc8929a736a1fa5"
server_key_iv: "96e6d6377b561f8d8f40f4e07916c9d9"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.726846 28447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:33.749814 28447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:33.752674 28447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:33.752826 28447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:33.753304 28447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:33.754055 28447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:33.754154 28447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.754220 28447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:33.754245 28447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.763269 28447 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:44631
I20260927 00:37:33.763295 28575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:44631 every 8 connection(s)
I20260927 00:37:33.764082 28447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:33.770258 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28447
I20260927 00:37:33.770421 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:33.770800 10287 external_mini_cluster.cc:1428] Setting key 09493ef79b1223bd75e2b8b05940358f
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:33.777935 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:33.765754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52747 (local address 127.10.11.254:42061)
0927 00:37:33.765893 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:33.765897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:33.766773 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:37:33.768033 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:33.768036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:33.768081 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:33.768165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:33.769279 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.769856 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.771478 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.771772 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.771858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:33.774460 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:33.774463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:33.774466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:33.774495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:33.775988 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.776479 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.776482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.776484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.776569 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.776907 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.776910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.776911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.777083 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:33.777187 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:33.777653 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 00:37:33.777774 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:37:33.778684 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:33.766188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42061 (local address 127.10.11.193:52747)
0927 00:37:33.766625 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:33.766667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:33.767842 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:33.768330 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:33.768337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:33.768695 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:33.769123 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:33.769136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.770037 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.770044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:33.771368 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:33.771378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.771934 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.771939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:33.772120 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:33.772671 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:33.772686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:33.774307 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:33.776103 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:33.776108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:33.776122 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:33.776371 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:33.776668 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:33.776671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:33.776673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:33.776807 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:33.777185 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:33.777190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:33.777425 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 00:37:33.777598 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:37:33.777895 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:37:33.779897 28576 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42061
I20260927 00:37:33.780184 28576 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:33.780742 28576 heartbeater.cc:511] Master 127.10.11.254:42061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:37:33.782446 28385 ts_manager.cc:194] Registered new tserver with Master: 9ce13fb1fa5244238cec886a0d2260c3 (127.10.11.193:44631)
I20260927 00:37:33.783855 28385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:52747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:37:33.796379 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:33.901288 28581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:33.901536 28581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:33.901597 28581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:33.906261 28581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:33.906309 28581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:33.906364 28581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:33.911456 28581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:33.912269 28581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:33.913125 28581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:33.921905 28589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:33.921906 28587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:33.921896 28586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:33.922164 28581 server_base.cc:1034] running on GCE node
I20260927 00:37:33.922605 28581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:33.923192 28581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:33.924413 28581 hybrid_clock.cc:648] HybridClock initialized: now 1790469453924394 us; error 34 us; skew 500 ppm
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:33.926959 28581 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:37:33.927783 28581 webserver.cc:460] Webserver started at http://127.10.11.194:43937/ using document root <none> and password file <none>
I20260927 00:37:33.928292 28581 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:33.928377 28581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:33.928588 28581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:33.930560 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "d9fd067111554c5abb1dbfc80e85742c"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2ed61f7e0bdb7e91481c28b7199e33b9"
server_key_iv: "b2ced40fd2b638377a29df466f925a17"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.931071 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "d9fd067111554c5abb1dbfc80e85742c"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2ed61f7e0bdb7e91481c28b7199e33b9"
server_key_iv: "b2ced40fd2b638377a29df466f925a17"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.934418 28581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:33.936537 28596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.937669 28581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:33.937791 28581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "d9fd067111554c5abb1dbfc80e85742c"
format_stamp: "Formatted at 2026-09-27 00:37:33 on dist-test-slave-6gxc"
server_key: "2ed61f7e0bdb7e91481c28b7199e33b9"
server_key_iv: "b2ced40fd2b638377a29df466f925a17"
server_key_version: "encryptionkey@0"
I20260927 00:37:33.937887 28581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:33.959908 28581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:33.962925 28581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:33.963106 28581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:33.963621 28581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:33.964366 28581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:33.964435 28581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.964502 28581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:33.964548 28581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:33.972754 28581 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:34709
I20260927 00:37:33.972776 28709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:34709 every 8 connection(s)
I20260927 00:37:33.973552 28581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:33.981743 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28581
I20260927 00:37:33.981848 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
Sep 27 00:37:33 dist-test-slave-6gxc krb5kdc[28340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469453, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:33.982167 10287 external_mini_cluster.cc:1428] Setting key 04fc355421f154bb6236029d33b41993
I20260927 00:37:33.986488 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:33.975195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:40177 (local address 127.10.11.254:42061)
0927 00:37:33.975322 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:33.975325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:33.976081 (+ 756us) server_negotiation.cc:407] Connection header received
0927 00:37:33.977299 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:33.977303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:33.977357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:33.977446 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:33.978494 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.979017 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.980102 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.980325 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.980445 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:33.982762 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:33.982766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:33.982768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:33.982794 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:33.984522 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.984988 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.984995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.984999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.985092 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:33.985431 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:33.985436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:33.985438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:33.985621 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:33.985731 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:33.986212 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 00:37:33.986327 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:37:33.987185 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:33.975513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42061 (local address 127.10.11.194:40177)
0927 00:37:33.975935 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:33.975973 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:33.977102 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:33.977606 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:33.977613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:33.977906 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:33.978354 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:33.978367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.979139 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.979143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:33.979977 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:33.979986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:33.980478 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:33.980483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:33.980635 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:33.981048 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:33.981062 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:33.982582 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:33.984637 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:33.984641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:33.984652 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:33.984891 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:33.985182 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:33.985184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:33.985185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:33.985303 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:33.985729 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:33.985735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:33.985949 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:37:33.986155 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 00:37:33.986445 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260927 00:37:33.988250 28710 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42061
I20260927 00:37:33.988471 28710 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:33.988904 28710 heartbeater.cc:511] Master 127.10.11.254:42061 requested a full tablet report, sending...
I20260927 00:37:33.989872 28385 ts_manager.cc:194] Registered new tserver with Master: d9fd067111554c5abb1dbfc80e85742c (127.10.11.194:34709)
I20260927 00:37:33.990696 28385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:40177
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:37:34.008478 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:34.113425 28715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:34.113639 28715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:34.113679 28715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:34.117635 28715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:34.117684 28715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:34.117779 28715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:34.122875 28715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:34.123759 28715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:34.124655 28715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:34.133682 28721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:34.133775 28723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:34.133682 28720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:34.133868 28715 server_base.cc:1034] running on GCE node
I20260927 00:37:34.134331 28715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:34.134884 28715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:34.136061 28715 hybrid_clock.cc:648] HybridClock initialized: now 1790469454136034 us; error 41 us; skew 500 ppm
Sep 27 00:37:34 dist-test-slave-6gxc krb5kdc[28340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:34.138588 28715 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:37:34.139396 28715 webserver.cc:460] Webserver started at http://127.10.11.195:38653/ using document root <none> and password file <none>
I20260927 00:37:34.139919 28715 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:34.140017 28715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:34.140232 28715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:34.142154 28715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "58d73c77c032466787feff14be2f07cd"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "e3fdade5514450a884b11be2a69afa9c"
server_key_iv: "aea4e0a86b9c12080723a7b4162b6617"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.142690 28715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "58d73c77c032466787feff14be2f07cd"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "e3fdade5514450a884b11be2a69afa9c"
server_key_iv: "aea4e0a86b9c12080723a7b4162b6617"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.145931 28715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:37:34.148137 28730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:34.149235 28715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:34.149362 28715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "58d73c77c032466787feff14be2f07cd"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "e3fdade5514450a884b11be2a69afa9c"
server_key_iv: "aea4e0a86b9c12080723a7b4162b6617"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.149462 28715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:34.173511 28715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:34.176298 28715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:34.176506 28715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:34.177026 28715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:34.177758 28715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:34.177829 28715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:34.177899 28715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:34.177950 28715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:34.186678 28715 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:46333
I20260927 00:37:34.186704 28843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:46333 every 8 connection(s)
I20260927 00:37:34.187471 28715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:34.193773 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28715
I20260927 00:37:34.193908 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:34.194252 10287 external_mini_cluster.cc:1428] Setting key c9d787cf7b6e7a82ae9b31c88cb0d0b6
Sep 27 00:37:34 dist-test-slave-6gxc krb5kdc[28340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:34.200706 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.189067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:56685 (local address 127.10.11.254:42061)
0927 00:37:34.189217 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.189221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.190031 (+ 810us) server_negotiation.cc:407] Connection header received
0927 00:37:34.191338 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.191340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.191386 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.191458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:34.192536 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.193053 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.194652 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.194947 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.195030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.197476 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:34.197479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:34.197481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:34.197502 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:34.199034 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:34.199473 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:34.199476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:34.199477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:34.199549 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:34.199826 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:34.199828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:34.199829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:34.199986 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:34.200059 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.200490 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.200581 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:37:34.201515 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.189447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:42061 (local address 127.10.11.195:56685)
0927 00:37:34.189881 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.189928 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.191151 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.191610 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.191616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.191942 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:34.192375 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.192388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.193182 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.193185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.194494 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.194506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.195080 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.195087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.195246 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.195888 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:34.195903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:34.197333 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:34.199131 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:34.199135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:34.199160 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:34.199389 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:34.199629 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:34.199631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:34.199632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:34.199749 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:34.200068 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:34.200072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:34.200297 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.200441 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.200667 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260927 00:37:34.202557 28844 heartbeater.cc:347] Connected to a master server at 127.10.11.254:42061
I20260927 00:37:34.202775 28844 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:34.203238 28844 heartbeater.cc:511] Master 127.10.11.254:42061 requested a full tablet report, sending...
I20260927 00:37:34.204231 28385 ts_manager.cc:194] Registered new tserver with Master: 58d73c77c032466787feff14be2f07cd (127.10.11.195:46333)
I20260927 00:37:34.204813 28385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:56685
I20260927 00:37:34.207996 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:34.216660 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.209860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.10.11.254:42061)
0927 00:37:34.209967 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.209971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.210173 (+ 202us) server_negotiation.cc:407] Connection header received
0927 00:37:34.210313 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.210315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.210350 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.210413 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:34.211027 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.211490 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.212263 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.212455 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.212525 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.213860 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:34.213863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:34.213865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:34.213884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:34.215278 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:34.215691 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:34.215693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:34.215694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:34.215761 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:34.216012 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:34.216014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:34.216015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:34.216176 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:34.216282 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.216479 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.216546 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:37:34.219214 28385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:34.222005 28385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:34.235630 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.230429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.10.11.193:44631)
0927 00:37:34.230818 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.230833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.230857 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:37:34.230935 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.230940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.231149 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.231327 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:34.231639 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.232567 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.234216 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.234981 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235104 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.235230 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.235402 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.235489 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":69,"threads_started":1}
I20260927 00:37:34.235630 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.229772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44631 (local address 127.0.0.1:42188)
0927 00:37:34.230553 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.230597 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.230719 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.231332 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.231337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.231363 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:34.231493 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.231502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.232686 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.232693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.234058 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.234069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.235096 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.235203 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.235239 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.235372 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.235488 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":51,"thread_start_us":74,"threads_started":1}
I20260927 00:37:34.236235 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.230040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.10.11.195:46333)
0927 00:37:34.230331 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.230343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.231136 (+ 793us) server_negotiation.cc:407] Connection header received
0927 00:37:34.231213 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.231221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.231389 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.231521 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:34.231739 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.232860 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.234336 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.234988 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235094 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.235225 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.235984 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.236078 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
I20260927 00:37:34.236371 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.230508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:34709 (local address 127.0.0.1:53782)
0927 00:37:34.231490 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.231503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.231664 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.232144 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.232147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.232165 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:34.232258 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.232264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.233470 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.233476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.234670 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.234681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.235955 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.236055 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.236076 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.236166 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.236226 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260927 00:37:34.236438 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.230932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.10.11.194:34709)
0927 00:37:34.231363 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.231374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.231577 (+ 203us) server_negotiation.cc:407] Connection header received
0927 00:37:34.231825 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.231831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.232002 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.232145 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:34.232378 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.233317 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.234801 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.235657 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235771 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.235898 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.236219 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.236309 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":97,"threads_started":1}
I20260927 00:37:34.238281 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.229903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:46333 (local address 127.0.0.1:52968)
0927 00:37:34.230940 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.230952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.231030 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.231504 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.231507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.231524 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:34.231622 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.231629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.232994 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.232998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.234225 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.234238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.235287 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.235290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.235373 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.235411 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.235487 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.235557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260927 00:37:34.238709 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8b1503285448369d822295129ebd74 (DEFAULT_TABLE table=test-table [id=d2aef5b8133f49df9951644ab3f0b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:34.239416 28644 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8b1503285448369d822295129ebd74 (DEFAULT_TABLE table=test-table [id=d2aef5b8133f49df9951644ab3f0b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:34.239838 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8b1503285448369d822295129ebd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:34.240356 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8b1503285448369d822295129ebd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:34.243122 28778 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8b1503285448369d822295129ebd74 (DEFAULT_TABLE table=test-table [id=d2aef5b8133f49df9951644ab3f0b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:34.244056 28778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8b1503285448369d822295129ebd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:34.247165 28861 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Bootstrap starting.
I20260927 00:37:34.248594 28862 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Bootstrap starting.
I20260927 00:37:34.249476 28861 tablet_bootstrap.cc:654] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:34.250672 28863 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Bootstrap starting.
I20260927 00:37:34.250725 28862 tablet_bootstrap.cc:654] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:34.251008 28861 log.cc:824] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:34.252238 28862 log.cc:824] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:34.252554 28863 tablet_bootstrap.cc:654] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:34.253129 28861 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: No bootstrap required, opened a new log
I20260927 00:37:34.253319 28861 ts_tablet_manager.cc:1397] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:37:34.253706 28863 log.cc:824] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:34.254433 28862 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: No bootstrap required, opened a new log
I20260927 00:37:34.254608 28862 ts_tablet_manager.cc:1397] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:37:34.255283 28863 tablet_bootstrap.cc:492] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: No bootstrap required, opened a new log
I20260927 00:37:34.255457 28863 ts_tablet_manager.cc:1397] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:37:34.257151 28861 raft_consensus.cc:348] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.257421 28861 raft_consensus.cc:374] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:34.257469 28861 raft_consensus.cc:729] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce13fb1fa5244238cec886a0d2260c3, State: Initialized, Role: FOLLOWER
I20260927 00:37:34.257941 28861 consensus_queue.cc:260] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.258237 28863 raft_consensus.cc:348] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.258279 28862 raft_consensus.cc:348] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.258466 28863 raft_consensus.cc:374] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:34.258502 28862 raft_consensus.cc:374] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:34.258522 28863 raft_consensus.cc:729] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d73c77c032466787feff14be2f07cd, State: Initialized, Role: FOLLOWER
I20260927 00:37:34.258543 28862 raft_consensus.cc:729] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fd067111554c5abb1dbfc80e85742c, State: Initialized, Role: FOLLOWER
I20260927 00:37:34.258927 28861 ts_tablet_manager.cc:1428] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 00:37:34.258929 28862 consensus_queue.cc:260] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.258924 28863 consensus_queue.cc:260] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.259284 28576 heartbeater.cc:503] Master 127.10.11.254:42061 was elected leader, sending a full tablet report...
I20260927 00:37:34.259554 28844 heartbeater.cc:503] Master 127.10.11.254:42061 was elected leader, sending a full tablet report...
I20260927 00:37:34.259816 28862 ts_tablet_manager.cc:1428] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:37:34.259851 28863 ts_tablet_manager.cc:1428] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 00:37:34.260130 28710 heartbeater.cc:503] Master 127.10.11.254:42061 was elected leader, sending a full tablet report...
W20260927 00:37:34.265105 28577 tablet.cc:2369] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:37:34.438326 28845 tablet.cc:2369] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:34.463655 28869 raft_consensus.cc:482] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:34.463866 28869 raft_consensus.cc:504] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.464986 28869 leader_election.cc:288] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ce13fb1fa5244238cec886a0d2260c3 (127.10.11.193:44631), d9fd067111554c5abb1dbfc80e85742c (127.10.11.194:34709)
I20260927 00:37:34.468588 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.465341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:33879 (local address 127.10.11.194:34709)
0927 00:37:34.465479 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.465483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.465501 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:37:34.465660 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.465664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.465720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.465825 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:34.466085 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.466623 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.467658 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.468052 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.468194 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.468246 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.468416 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.468486 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:37:34.468590 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.465229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:34709 (local address 127.10.11.195:33879)
0927 00:37:34.465376 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.465392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.465512 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.465834 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.465837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.465856 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:34.465978 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.465983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.466741 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.466745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.467540 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.467548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.468181 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.468184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.468285 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.468311 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.468398 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.468463 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:37:34.468775 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.465424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:59779 (local address 127.10.11.193:44631)
0927 00:37:34.465555 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.465561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.465815 (+ 254us) server_negotiation.cc:407] Connection header received
0927 00:37:34.465959 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.465962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.466013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.466095 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:34.466392 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.466879 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.467874 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.468265 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.468398 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.468447 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.468607 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.468671 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:37:34.468792 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.465229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44631 (local address 127.10.11.195:59779)
0927 00:37:34.465738 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.465748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:34.465849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:34.466127 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:34.466130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:34.466144 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:34.466273 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.466278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.466988 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.466994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.467741 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:34.467749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.468383 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.468386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:34.468479 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.468502 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:37:34.468591 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:37:34.468652 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":244,"thread_start_us":113,"threads_started":1}
I20260927 00:37:34.469300 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8b1503285448369d822295129ebd74" candidate_uuid: "58d73c77c032466787feff14be2f07cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fd067111554c5abb1dbfc80e85742c" is_pre_election: true
I20260927 00:37:34.469340 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8b1503285448369d822295129ebd74" candidate_uuid: "58d73c77c032466787feff14be2f07cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce13fb1fa5244238cec886a0d2260c3" is_pre_election: true
I20260927 00:37:34.469609 28530 raft_consensus.cc:2454] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d73c77c032466787feff14be2f07cd in term 0.
I20260927 00:37:34.469609 28664 raft_consensus.cc:2454] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d73c77c032466787feff14be2f07cd in term 0.
I20260927 00:37:34.470077 28733 leader_election.cc:302] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d73c77c032466787feff14be2f07cd, 9ce13fb1fa5244238cec886a0d2260c3; no voters:
I20260927 00:37:34.470381 28869 raft_consensus.cc:2781] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:34.470472 28869 raft_consensus.cc:482] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:34.470548 28869 raft_consensus.cc:3037] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:34.471517 28869 raft_consensus.cc:504] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
I20260927 00:37:34.471834 28869 leader_election.cc:288] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [CANDIDATE]: Term 1 election: Requested vote from peers 9ce13fb1fa5244238cec886a0d2260c3 (127.10.11.193:44631), d9fd067111554c5abb1dbfc80e85742c (127.10.11.194:34709)
I20260927 00:37:34.472211 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8b1503285448369d822295129ebd74" candidate_uuid: "58d73c77c032466787feff14be2f07cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce13fb1fa5244238cec886a0d2260c3"
I20260927 00:37:34.472239 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8b1503285448369d822295129ebd74" candidate_uuid: "58d73c77c032466787feff14be2f07cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fd067111554c5abb1dbfc80e85742c"
I20260927 00:37:34.472332 28664 raft_consensus.cc:3037] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:34.472335 28530 raft_consensus.cc:3037] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:34.473330 28664 raft_consensus.cc:2454] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d73c77c032466787feff14be2f07cd in term 1.
I20260927 00:37:34.473333 28530 raft_consensus.cc:2454] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d73c77c032466787feff14be2f07cd in term 1.
I20260927 00:37:34.473631 28734 leader_election.cc:302] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d73c77c032466787feff14be2f07cd, d9fd067111554c5abb1dbfc80e85742c; no voters:
I20260927 00:37:34.473829 28869 raft_consensus.cc:2781] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:34.474020 28869 raft_consensus.cc:686] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 1 LEADER]: Becoming Leader. State: Replica: 58d73c77c032466787feff14be2f07cd, State: Running, Role: LEADER
I20260927 00:37:34.474355 28869 consensus_queue.cc:237] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } }
W20260927 00:37:34.474656 28711 tablet.cc:2369] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:34.477166 28383 catalog_manager.cc:5495] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d73c77c032466787feff14be2f07cd (127.10.11.195). New cstate: current_term: 1 leader_uuid: "58d73c77c032466787feff14be2f07cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d73c77c032466787feff14be2f07cd" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 46333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 } health_report { overall_health: UNKNOWN } } }
I20260927 00:37:34.525074 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.521928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.10.11.195:46333)
0927 00:37:34.522063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.522067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.522123 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:37:34.522185 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.522187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.522237 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.522311 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:34.522528 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.523007 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.523794 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.523996 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.524075 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.524353 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:34.524712 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:34.524816 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.524927 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.524959 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:37:34.530555 28530 raft_consensus.cc:1261] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 1 FOLLOWER]: Refusing update from remote peer 58d73c77c032466787feff14be2f07cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:34.530565 28664 raft_consensus.cc:1261] T 7e8b1503285448369d822295129ebd74 P d9fd067111554c5abb1dbfc80e85742c [term 1 FOLLOWER]: Refusing update from remote peer 58d73c77c032466787feff14be2f07cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:34.531193 28869 consensus_queue.cc:1035] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fd067111554c5abb1dbfc80e85742c" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:34.531401 28871 consensus_queue.cc:1035] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ce13fb1fa5244238cec886a0d2260c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:34.537873 28874 mvcc.cc:205] Tried to move back new op lower bound from 7333762885749985280 to 7333762885529518080. Current Snapshot: MvccSnapshot[applied={T|T < 7333762885749985280}]
I20260927 00:37:34.538388 28876 mvcc.cc:205] Tried to move back new op lower bound from 7333762885749985280 to 7333762885529518080. Current Snapshot: MvccSnapshot[applied={T|T < 7333762885749985280}]
I20260927 00:37:34.541633 28875 mvcc.cc:205] Tried to move back new op lower bound from 7333762885749985280 to 7333762885529518080. Current Snapshot: MvccSnapshot[applied={T|T < 7333762885749985280 or (T in {7333762885749985280})}]
I20260927 00:37:34.545038 28384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:34.545190 28384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:34.547403 28384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef19aedfe57841fa8c4528378b7eaf95: Sending DeleteTablet for 3 replicas of tablet 7e8b1503285448369d822295129ebd74
I20260927 00:37:34.548070 28510 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8b1503285448369d822295129ebd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:34 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:42188
I20260927 00:37:34.548142 28644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8b1503285448369d822295129ebd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:34 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:53782
I20260927 00:37:34.548559 28884 tablet_replica.cc:330] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3: stopping tablet replica
I20260927 00:37:34.548648 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8b1503285448369d822295129ebd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:34 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:52968
I20260927 00:37:34.548916 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28447
I20260927 00:37:34.548959 28884 raft_consensus.cc:2229] T 7e8b1503285448369d822295129ebd74 P 9ce13fb1fa5244238cec886a0d2260c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:34.549666 28886 tablet_replica.cc:330] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: stopping tablet replica
I20260927 00:37:34.549963 28886 raft_consensus.cc:2229] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:37:34.550266 28886 raft_consensus.cc:2258] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:34.551424 28886 ts_tablet_manager.cc:1905] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:34.553599 28886 ts_tablet_manager.cc:1918] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:34.553704 28886 log.cc:1196] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/7e8b1503285448369d822295129ebd74
I20260927 00:37:34.554005 28886 ts_tablet_manager.cc:1939] T 7e8b1503285448369d822295129ebd74 P 58d73c77c032466787feff14be2f07cd: Deleting consensus metadata
W20260927 00:37:34.554881 28371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:44631 (error 108)
I20260927 00:37:34.554983 28370 catalog_manager.cc:4841] TS 58d73c77c032466787feff14be2f07cd (127.10.11.195:46333): tablet 7e8b1503285448369d822295129ebd74 (table test-table [id=d2aef5b8133f49df9951644ab3f0b78e]) successfully deleted
I20260927 00:37:34.555105 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28581
I20260927 00:37:34.555859 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.555462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:44631 (local address 127.0.0.1:42204)
0927 00:37:34.555745 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.555797 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260927 00:37:34.555994 28371 catalog_manager.cc:4584] TS 9ce13fb1fa5244238cec886a0d2260c3 (127.10.11.193:44631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e8b1503285448369d822295129ebd74: Network error: Client connection negotiation failed: client connection to 127.10.11.193:44631: connect: Connection refused (error 111)
I20260927 00:37:34.562064 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28715
I20260927 00:37:34.563336 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.562282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:34709 (local address 127.0.0.1:53788)
0927 00:37:34.563186 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:34.563249 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.194:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812}
I20260927 00:37:34.567624 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28353
2026-09-27T00:37:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2422 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:34Z Disabled control of system clock
I20260927 00:37:34.596244 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36663
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:34.704722 28891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:34.704950 28891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:34.704983 28891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:34.708809 28891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:34.708854 28891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:34.708869 28891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:34.708885 28891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:34.708900 28891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:34.713455 28891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:34.714349 28891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:34.715559 28891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:34.723822 28897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:34.723846 28899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:34.723858 28896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:34.723920 28891 server_base.cc:1034] running on GCE node
I20260927 00:37:34.724620 28891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:34.725683 28891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:34.726854 28891 hybrid_clock.cc:648] HybridClock initialized: now 1790469454726825 us; error 39 us; skew 500 ppm
I20260927 00:37:34.728646 28891 webserver.cc:460] Webserver started at http://127.10.11.254:38755/ using document root <none> and password file <none>
I20260927 00:37:34.729195 28891 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:34.729257 28891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:34.729488 28891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:34.731482 28891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "b9da864827928cc8703e92dac952b95d"
server_key_iv: "ce30cb0300643915241edbb98d5cdaeb"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.732028 28891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "b9da864827928cc8703e92dac952b95d"
server_key_iv: "ce30cb0300643915241edbb98d5cdaeb"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.735496 28891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:37:34.737715 28905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:34.739209 28891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:34.739301 28891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "b9da864827928cc8703e92dac952b95d"
server_key_iv: "ce30cb0300643915241edbb98d5cdaeb"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.739418 28891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:34.764348 28891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:34.764947 28891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:34.765139 28891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:34.771433 28891 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:36663
I20260927 00:37:34.771463 28957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:36663 every 8 connection(s)
I20260927 00:37:34.772338 28891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:34.775471 28958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:34.781102 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: Bootstrap starting.
I20260927 00:37:34.781520 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28891
I20260927 00:37:34.781653 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:34.781944 10287 external_mini_cluster.cc:1428] Setting key 93f0ac620db8a6e25a14b8f0e3789377
I20260927 00:37:34.783901 28958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:34.784828 28958 log.cc:824] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:34.787075 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: No bootstrap required, opened a new log
I20260927 00:37:34.791679 28958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } }
I20260927 00:37:34.791954 28958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:34.792025 28958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1cf85a7c07422fa64de5b9f46c2e9b, State: Initialized, Role: FOLLOWER
I20260927 00:37:34.792510 28958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } }
I20260927 00:37:34.792675 28958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:34.792681 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:34.783327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.10.11.254:36663)
0927 00:37:34.784009 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:34.784031 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:34.784078 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:37:34.784969 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:34.784993 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:34.785476 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:34.785857 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:34.786310 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.787888 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.788764 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:34.789525 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:34.789696 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:34.790484 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:34.790493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:34.790513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:34.791089 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:34.791162 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:34.791189 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:34.791297 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:34.791433 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 00:37:34.791702 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
I20260927 00:37:34.792768 28958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:34.792860 28958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:34.793727 28958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } }
I20260927 00:37:34.794082 28958 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1cf85a7c07422fa64de5b9f46c2e9b; no voters:
I20260927 00:37:34.794482 28958 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:34.794585 28963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:34.794791 28963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [term 1 LEADER]: Becoming Leader. State: Replica: 2e1cf85a7c07422fa64de5b9f46c2e9b, State: Running, Role: LEADER
I20260927 00:37:34.795086 28963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } }
I20260927 00:37:34.795745 28958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:34.796764 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1cf85a7c07422fa64de5b9f46c2e9b. Latest consensus state: current_term: 1 leader_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } } }
I20260927 00:37:34.796901 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:34.797219 28964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1cf85a7c07422fa64de5b9f46c2e9b" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36663 } } }
I20260927 00:37:34.797307 28972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:34.797333 28964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:34.800915 28972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:34.806007 28972 catalog_manager.cc:1347] Generated new cluster ID: abe07e783a1348b7b0a883f5eab5ec1a
I20260927 00:37:34.806072 28972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:34.821825 28972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:34.823041 28972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:34.837091 28972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: Generated new TSK 0
I20260927 00:37:34.837909 28972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:34.847033 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--builtin_ntp_servers=127.10.11.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:34.957093 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:34.957321 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:34.957357 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:34.961292 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:34.961335 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:34.961386 28982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:34.965966 28982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:34.966910 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:34.968183 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:34.977192 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:34.977262 28982 server_base.cc:1034] running on GCE node
W20260927 00:37:34.977193 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:34.977217 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:34.977944 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:34.978546 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:34.979714 28982 hybrid_clock.cc:648] HybridClock initialized: now 1790469454979702 us; error 25 us; skew 500 ppm
I20260927 00:37:34.981447 28982 webserver.cc:460] Webserver started at http://127.10.11.193:40719/ using document root <none> and password file <none>
I20260927 00:37:34.981992 28982 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:34.982049 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:34.982282 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:34.984182 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "098ba9ed023d4c0c87e5be2bb90dc33f"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "c98ebe815e1cb82522466f97a9752804"
server_key_iv: "50d4c8cde46ad092eb9e1e119ee48c8e"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.984732 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "098ba9ed023d4c0c87e5be2bb90dc33f"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "c98ebe815e1cb82522466f97a9752804"
server_key_iv: "50d4c8cde46ad092eb9e1e119ee48c8e"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.988351 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:37:34.990741 28996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:34.991920 28982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:34.992045 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "098ba9ed023d4c0c87e5be2bb90dc33f"
format_stamp: "Formatted at 2026-09-27 00:37:34 on dist-test-slave-6gxc"
server_key: "c98ebe815e1cb82522466f97a9752804"
server_key_iv: "50d4c8cde46ad092eb9e1e119ee48c8e"
server_key_version: "encryptionkey@0"
I20260927 00:37:34.992189 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:35.007383 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:35.007978 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:35.008174 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:35.008718 28982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:35.009575 28982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:35.009621 28982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.009686 28982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:35.009717 28982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.018596 28982 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:45807
I20260927 00:37:35.018622 29109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:45807 every 8 connection(s)
I20260927 00:37:35.019412 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:35.022823 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 28982
I20260927 00:37:35.022982 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:35.023293 10287 external_mini_cluster.cc:1428] Setting key e3a494ab7436920f086c45bd835f022e
I20260927 00:37:35.025520 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--builtin_ntp_servers=127.10.11.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:35.029819 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.021249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:53983 (local address 127.10.11.254:36663)
0927 00:37:35.021384 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.021388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.022456 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 00:37:35.024085 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.024088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.024147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.024236 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:35.025324 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.026015 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.027549 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.027948 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.028076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.029228 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:35.029235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:35.029241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:35.029258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:35.029284 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:35.029298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:35.029387 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.029536 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.029609 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:37:35.030560 29112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.021649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36663 (local address 127.10.11.193:53983)
0927 00:37:35.022318 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.022356 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.023864 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.024425 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.024433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.024513 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:35.025163 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.025181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.026241 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.026248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.027421 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.027430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.028137 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.028141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.028277 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.028934 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:35.028936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:35.028974 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:35.029034 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:35.029394 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:35.029401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:35.029405 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.029504 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.029727 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260927 00:37:35.031824 29110 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36663
I20260927 00:37:35.032083 29110 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:35.032716 29110 heartbeater.cc:511] Master 127.10.11.254:36663 requested a full tablet report, sending...
I20260927 00:37:35.034282 28922 ts_manager.cc:194] Registered new tserver with Master: 098ba9ed023d4c0c87e5be2bb90dc33f (127.10.11.193:45807)
I20260927 00:37:35.035372 28922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.193:53983
W20260927 00:37:35.132987 29113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:35.133201 29113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:35.133270 29113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:35.137122 29113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:35.137166 29113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:35.137253 29113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:35.141822 29113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:35.142802 29113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:35.144060 29113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:35.153047 29121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:35.153059 29118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:35.153087 29119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:35.153321 29113 server_base.cc:1034] running on GCE node
I20260927 00:37:35.153751 29113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:35.154361 29113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:35.155517 29113 hybrid_clock.cc:648] HybridClock initialized: now 1790469455155498 us; error 29 us; skew 500 ppm
I20260927 00:37:35.157136 29113 webserver.cc:460] Webserver started at http://127.10.11.194:33379/ using document root <none> and password file <none>
I20260927 00:37:35.157660 29113 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:35.157716 29113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:35.157941 29113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:35.159870 29113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "610cb503cca64a918af8dd83f47002f2"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "0a499dca56176542cf303f3133b80bee"
server_key_iv: "d8a9744180de8cd233a26343f711885a"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.160388 29113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "610cb503cca64a918af8dd83f47002f2"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "0a499dca56176542cf303f3133b80bee"
server_key_iv: "d8a9744180de8cd233a26343f711885a"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.163785 29113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:37:35.166079 29127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.167243 29113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:35.167364 29113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "610cb503cca64a918af8dd83f47002f2"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "0a499dca56176542cf303f3133b80bee"
server_key_iv: "d8a9744180de8cd233a26343f711885a"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.167467 29113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:35.191910 29113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:35.192503 29113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:35.192687 29113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:35.193207 29113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:35.194077 29113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:35.194186 29113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.194253 29113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:35.194299 29113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.202692 29113 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:44273
I20260927 00:37:35.202730 29240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:44273 every 8 connection(s)
I20260927 00:37:35.203482 29113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:35.211660 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.205076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:58695 (local address 127.10.11.254:36663)
0927 00:37:35.205255 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.205260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.206299 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 00:37:35.207577 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.207580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.207629 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.207701 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:35.208485 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.208989 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.210012 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.210230 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.210304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.211288 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:35.211291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:35.211293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:35.211302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:35.211317 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:35.211324 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:35.211375 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.211503 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.211548 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:37:35.212035 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29113
I20260927 00:37:35.212167 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:37:35.212457 10287 external_mini_cluster.cc:1428] Setting key 2063b7e07c3d4f68e51a151b199221c4
I20260927 00:37:35.212538 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.205473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36663 (local address 127.10.11.194:58695)
0927 00:37:35.206160 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.206205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.207392 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.207851 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.207857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.207930 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:35.208372 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.208383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.209091 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.209093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.209913 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.209921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.210346 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.210348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.210475 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.211077 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:35.211079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:35.211114 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:35.211170 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:35.211392 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:35.211399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:35.211402 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.211488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.211689 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260927 00:37:35.213604 29241 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36663
I20260927 00:37:35.213820 29241 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:35.214299 29241 heartbeater.cc:511] Master 127.10.11.254:36663 requested a full tablet report, sending...
I20260927 00:37:35.214532 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--builtin_ntp_servers=127.10.11.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:35.215294 28921 ts_manager.cc:194] Registered new tserver with Master: 610cb503cca64a918af8dd83f47002f2 (127.10.11.194:44273)
I20260927 00:37:35.215849 28921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.194:58695
W20260927 00:37:35.320564 29244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:35.320803 29244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:35.320837 29244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:35.324738 29244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:35.324783 29244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:35.324833 29244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:35.329324 29244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:35.330312 29244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:35.331509 29244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:35.340485 29252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:35.340651 29244 server_base.cc:1034] running on GCE node
W20260927 00:37:35.340502 29250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:35.340528 29249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:35.341162 29244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:35.341696 29244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:35.342854 29244 hybrid_clock.cc:648] HybridClock initialized: now 1790469455342822 us; error 48 us; skew 500 ppm
I20260927 00:37:35.344480 29244 webserver.cc:460] Webserver started at http://127.10.11.195:38387/ using document root <none> and password file <none>
I20260927 00:37:35.345009 29244 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:35.345067 29244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:35.345270 29244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:35.347160 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "afb907eedb76c3ad27643614e1503a92"
server_key_iv: "f994e4f7902252ee7cd882f056da4758"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.347671 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "afb907eedb76c3ad27643614e1503a92"
server_key_iv: "f994e4f7902252ee7cd882f056da4758"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.351014 29244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:35.353211 29258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.354372 29244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:35.354494 29244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e"
format_stamp: "Formatted at 2026-09-27 00:37:35 on dist-test-slave-6gxc"
server_key: "afb907eedb76c3ad27643614e1503a92"
server_key_iv: "f994e4f7902252ee7cd882f056da4758"
server_key_version: "encryptionkey@0"
I20260927 00:37:35.354593 29244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:35.380990 29244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:35.381646 29244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:35.381835 29244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:35.382463 29244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:35.383360 29244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:35.383428 29244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.383499 29244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:35.383543 29244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:35.391954 29244 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:37877
I20260927 00:37:35.391983 29371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:37877 every 8 connection(s)
I20260927 00:37:35.392776 29244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:35.400803 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29244
I20260927 00:37:35.400926 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:35.401204 10287 external_mini_cluster.cc:1428] Setting key 85932dc4f15ce9870d4e1c3ecb7a10b8
I20260927 00:37:35.402004 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.394510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:55187 (local address 127.10.11.254:36663)
0927 00:37:35.394655 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.394658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.395649 (+ 991us) server_negotiation.cc:407] Connection header received
0927 00:37:35.397192 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.397196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.397260 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.397370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:35.398299 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.398825 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.399921 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.400158 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.400264 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.401471 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:35.401478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:35.401481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:35.401496 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:35.401527 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:35.401536 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:35.401595 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.401810 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.401868 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:37:35.402870 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.394850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36663 (local address 127.10.11.195:55187)
0927 00:37:35.395486 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.395542 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.396982 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.397535 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.397541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.397619 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:35.398158 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.398174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.398946 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.398949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.399800 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.399809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.400311 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.400315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.400472 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.401192 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:35.401193 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:35.401231 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:35.401316 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:35.401631 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:35.401639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:35.401647 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.401779 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.402028 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260927 00:37:35.403873 29372 heartbeater.cc:347] Connected to a master server at 127.10.11.254:36663
I20260927 00:37:35.404081 29372 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:35.404547 29372 heartbeater.cc:511] Master 127.10.11.254:36663 requested a full tablet report, sending...
I20260927 00:37:35.405433 28921 ts_manager.cc:194] Registered new tserver with Master: 23a9aa3f4a5f4761a1bdbdbb43a3997e (127.10.11.195:37877)
I20260927 00:37:35.406185 28921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.195:55187
I20260927 00:37:35.414970 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:35.420262 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.416764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.10.11.254:36663)
0927 00:37:35.416897 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.416901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.417068 (+ 167us) server_negotiation.cc:407] Connection header received
0927 00:37:35.417146 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.417148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.417182 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.417243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:35.417469 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.417956 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.418782 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.418989 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.419066 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.419790 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:35.419793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:35.419795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:35.419805 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:35.419821 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:35.419828 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:35.419938 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.420053 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.420112 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:37:35.423043 28921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:35.425820 28921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:35.438861 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.433631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:37877 (local address 127.0.0.1:36642)
0927 00:37:35.433953 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.434003 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.434094 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.434847 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.434855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.434881 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:35.435052 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.435060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.436296 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.436302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.437274 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.437283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.438341 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.438345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.438454 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.438489 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.438590 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.438712 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260927 00:37:35.438863 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.433785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.10.11.195:37877)
0927 00:37:35.434216 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.434230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.434257 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:37:35.434330 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.434335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.434531 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.434680 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:35.435171 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.436219 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.437406 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.438211 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.438318 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.438499 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.438614 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.438698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":138,"threads_started":1}
I20260927 00:37:35.439672 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.434367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:44273 (local address 127.0.0.1:49562)
0927 00:37:35.434926 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.434951 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.435033 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.435751 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.435754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.435771 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:35.435847 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.435852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.437075 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.437078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.438231 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.438241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.439270 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.439273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.439373 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.439397 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.439462 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.439529 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":95,"threads_started":1}
I20260927 00:37:35.439710 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.434551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.10.11.194:44273)
0927 00:37:35.435314 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.435326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.435363 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:37:35.435434 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.435439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.435620 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.435773 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:35.435942 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.436929 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.438355 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.439151 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.439274 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.439442 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.439490 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.439569 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":137,"threads_started":1}
I20260927 00:37:35.440274 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.434682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45807 (local address 127.0.0.1:41240)
0927 00:37:35.435437 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.435450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.435538 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.436051 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.436054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.436068 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:35.436143 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.436148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.437464 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.437467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.438611 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.438619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.439859 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.439866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.439966 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.439983 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.440087 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.440152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260927 00:37:35.440361 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.435061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.10.11.193:45807)
0927 00:37:35.435367 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.435378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.435537 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:37:35.435646 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.435650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.435808 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.435930 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:35.436273 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.437321 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.438731 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.439534 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.439625 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.439797 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.440118 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.440202 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260927 00:37:35.442072 29175 tablet_service.cc:1467] Processing CreateTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 (DEFAULT_TABLE table=test-table [id=336a21048bf34f198582b1f9f8a48f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:35.441953 29306 tablet_service.cc:1467] Processing CreateTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 (DEFAULT_TABLE table=test-table [id=336a21048bf34f198582b1f9f8a48f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:35.442708 29044 tablet_service.cc:1467] Processing CreateTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 (DEFAULT_TABLE table=test-table [id=336a21048bf34f198582b1f9f8a48f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:35.443068 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1380866eeb8a4bae8913fdd61f1495b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:35.443068 29175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1380866eeb8a4bae8913fdd61f1495b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:35.443572 29044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1380866eeb8a4bae8913fdd61f1495b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:35.450024 29389 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Bootstrap starting.
I20260927 00:37:35.450661 29391 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Bootstrap starting.
I20260927 00:37:35.450809 29390 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Bootstrap starting.
I20260927 00:37:35.451872 29389 tablet_bootstrap.cc:654] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:35.452579 29389 log.cc:824] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:35.452760 29390 tablet_bootstrap.cc:654] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:35.452760 29391 tablet_bootstrap.cc:654] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:35.453579 29391 log.cc:824] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:35.453579 29390 log.cc:824] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:35.454527 29389 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: No bootstrap required, opened a new log
I20260927 00:37:35.454721 29389 ts_tablet_manager.cc:1397] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 00:37:35.455503 29391 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: No bootstrap required, opened a new log
I20260927 00:37:35.455704 29391 ts_tablet_manager.cc:1397] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:37:35.456238 29390 tablet_bootstrap.cc:492] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: No bootstrap required, opened a new log
I20260927 00:37:35.456409 29390 ts_tablet_manager.cc:1397] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 00:37:35.459092 29391 raft_consensus.cc:348] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.459040 29389 raft_consensus.cc:348] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.459306 29391 raft_consensus.cc:374] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:35.459309 29389 raft_consensus.cc:374] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:35.459362 29391 raft_consensus.cc:729] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a9aa3f4a5f4761a1bdbdbb43a3997e, State: Initialized, Role: FOLLOWER
I20260927 00:37:35.459367 29389 raft_consensus.cc:729] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098ba9ed023d4c0c87e5be2bb90dc33f, State: Initialized, Role: FOLLOWER
I20260927 00:37:35.459774 29391 consensus_queue.cc:260] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.459798 29389 consensus_queue.cc:260] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.460407 29372 heartbeater.cc:503] Master 127.10.11.254:36663 was elected leader, sending a full tablet report...
I20260927 00:37:35.460493 29390 raft_consensus.cc:348] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.460704 29390 raft_consensus.cc:374] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:35.460741 29391 ts_tablet_manager.cc:1428] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:37:35.460757 29390 raft_consensus.cc:729] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610cb503cca64a918af8dd83f47002f2, State: Initialized, Role: FOLLOWER
I20260927 00:37:35.461112 29390 consensus_queue.cc:260] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.461272 29110 heartbeater.cc:503] Master 127.10.11.254:36663 was elected leader, sending a full tablet report...
I20260927 00:37:35.461675 29241 heartbeater.cc:503] Master 127.10.11.254:36663 was elected leader, sending a full tablet report...
I20260927 00:37:35.461998 29390 ts_tablet_manager.cc:1428] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:37:35.462808 29389 ts_tablet_manager.cc:1428] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260927 00:37:35.520458 29111 tablet.cc:2369] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:37:35.643700 29373 tablet.cc:2369] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:35.672410 29396 raft_consensus.cc:482] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:35.672683 29396 raft_consensus.cc:504] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.673859 29396 leader_election.cc:288] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a9aa3f4a5f4761a1bdbdbb43a3997e (127.10.11.195:37877), 610cb503cca64a918af8dd83f47002f2 (127.10.11.194:44273)
I20260927 00:37:35.678018 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.674678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:39177 (local address 127.10.11.194:44273)
0927 00:37:35.674799 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.674803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.674881 (+ 78us) server_negotiation.cc:407] Connection header received
0927 00:37:35.674998 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.675003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.675054 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.675134 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:35.675459 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.675977 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.677059 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.677452 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.677592 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.677645 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.677851 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.677919 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:37:35.678018 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.674086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:37877 (local address 127.10.11.193:42449)
0927 00:37:35.674386 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.674400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.674539 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.674815 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.674821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.674872 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:35.675036 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.675043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.675910 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.675913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.676790 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.676799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.677555 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.677559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.677667 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.677689 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.677760 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.677845 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260927 00:37:35.678180 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.674518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:44273 (local address 127.10.11.193:39177)
0927 00:37:35.674799 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.674813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:35.674896 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:35.675182 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:35.675185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:35.675204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:35.675318 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.675324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.676085 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.676087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.676823 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:35.676831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.677612 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.677617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:35.677707 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.677725 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:37:35.677824 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:37:35.677886 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260927 00:37:35.678107 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.674223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:42449 (local address 127.10.11.195:37877)
0927 00:37:35.674403 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.674411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.674467 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:37:35.674658 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.674662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.674714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.674791 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:35.675182 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.675795 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.676999 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.677420 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.677551 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.677610 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.677805 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.677877 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:37:35.678848 29326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1380866eeb8a4bae8913fdd61f1495b1" candidate_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" is_pre_election: true
I20260927 00:37:35.678821 29195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1380866eeb8a4bae8913fdd61f1495b1" candidate_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610cb503cca64a918af8dd83f47002f2" is_pre_election: true
I20260927 00:37:35.679152 29195 raft_consensus.cc:2454] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098ba9ed023d4c0c87e5be2bb90dc33f in term 0.
I20260927 00:37:35.679152 29326 raft_consensus.cc:2454] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098ba9ed023d4c0c87e5be2bb90dc33f in term 0.
I20260927 00:37:35.679617 28999 leader_election.cc:302] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098ba9ed023d4c0c87e5be2bb90dc33f, 610cb503cca64a918af8dd83f47002f2; no voters:
I20260927 00:37:35.679864 29396 raft_consensus.cc:2781] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:35.679963 29396 raft_consensus.cc:482] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:35.680009 29396 raft_consensus.cc:3037] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:35.681094 29396 raft_consensus.cc:504] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.681442 29396 leader_election.cc:288] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [CANDIDATE]: Term 1 election: Requested vote from peers 23a9aa3f4a5f4761a1bdbdbb43a3997e (127.10.11.195:37877), 610cb503cca64a918af8dd83f47002f2 (127.10.11.194:44273)
I20260927 00:37:35.681835 29326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1380866eeb8a4bae8913fdd61f1495b1" candidate_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e"
I20260927 00:37:35.682006 29195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1380866eeb8a4bae8913fdd61f1495b1" candidate_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610cb503cca64a918af8dd83f47002f2"
I20260927 00:37:35.682008 29326 raft_consensus.cc:3037] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:35.682142 29195 raft_consensus.cc:3037] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:35.683456 29326 raft_consensus.cc:2454] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098ba9ed023d4c0c87e5be2bb90dc33f in term 1.
I20260927 00:37:35.683456 29195 raft_consensus.cc:2454] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098ba9ed023d4c0c87e5be2bb90dc33f in term 1.
I20260927 00:37:35.683794 28999 leader_election.cc:302] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098ba9ed023d4c0c87e5be2bb90dc33f, 23a9aa3f4a5f4761a1bdbdbb43a3997e; no voters:
I20260927 00:37:35.683969 29396 raft_consensus.cc:2781] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:35.684212 29396 raft_consensus.cc:686] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [term 1 LEADER]: Becoming Leader. State: Replica: 098ba9ed023d4c0c87e5be2bb90dc33f, State: Running, Role: LEADER
I20260927 00:37:35.684568 29396 consensus_queue.cc:237] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } }
I20260927 00:37:35.687920 28920 catalog_manager.cc:5495] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f reported cstate change: term changed from 0 to 1, leader changed from <none> to 098ba9ed023d4c0c87e5be2bb90dc33f (127.10.11.193). New cstate: current_term: 1 leader_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098ba9ed023d4c0c87e5be2bb90dc33f" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 45807 } health_report { overall_health: HEALTHY } } }
W20260927 00:37:35.704507 29242 tablet.cc:2369] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:35.731074 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.727422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.10.11.193:45807)
0927 00:37:35.727573 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:35.727577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:35.727592 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:37:35.727688 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:35.727690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:35.727743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:35.727835 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:35.728100 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.728639 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.729495 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:35.729706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:35.729790 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:35.730073 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:35.730552 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:35.730666 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:35.730861 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:37:35.730904 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:37:35.736732 29326 raft_consensus.cc:1261] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 1 FOLLOWER]: Refusing update from remote peer 098ba9ed023d4c0c87e5be2bb90dc33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:35.736763 29195 raft_consensus.cc:1261] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 1 FOLLOWER]: Refusing update from remote peer 098ba9ed023d4c0c87e5be2bb90dc33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:35.737417 29396 consensus_queue.cc:1035] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a9aa3f4a5f4761a1bdbdbb43a3997e" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:35.737605 29400 consensus_queue.cc:1035] T 1380866eeb8a4bae8913fdd61f1495b1 P 098ba9ed023d4c0c87e5be2bb90dc33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "610cb503cca64a918af8dd83f47002f2" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 44273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:35.745318 29404 mvcc.cc:205] Tried to move back new op lower bound from 7333762890690560000 to 7333762890487582720. Current Snapshot: MvccSnapshot[applied={T|T < 7333762890690560000}]
I20260927 00:37:35.746325 29405 mvcc.cc:205] Tried to move back new op lower bound from 7333762890690560000 to 7333762890487582720. Current Snapshot: MvccSnapshot[applied={T|T < 7333762890690560000}]
I20260927 00:37:35.753808 28919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:35.753947 28919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:35.756488 28919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e1cf85a7c07422fa64de5b9f46c2e9b: Sending DeleteTablet for 3 replicas of tablet 1380866eeb8a4bae8913fdd61f1495b1
I20260927 00:37:35.757228 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:35 UTC) from {username='slave'} at 127.0.0.1:36642
I20260927 00:37:35.757351 29175 tablet_service.cc:1514] Processing DeleteTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:35 UTC) from {username='slave'} at 127.0.0.1:49562
I20260927 00:37:35.757711 29044 tablet_service.cc:1514] Processing DeleteTablet for tablet 1380866eeb8a4bae8913fdd61f1495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:35 UTC) from {username='slave'} at 127.0.0.1:41240
I20260927 00:37:35.757889 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28982
I20260927 00:37:35.758646 29413 tablet_replica.cc:330] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: stopping tablet replica
I20260927 00:37:35.758636 29414 tablet_replica.cc:330] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: stopping tablet replica
I20260927 00:37:35.758940 29413 raft_consensus.cc:2229] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:35.758940 29414 raft_consensus.cc:2229] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:35.759241 29413 raft_consensus.cc:2258] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:35.759255 29414 raft_consensus.cc:2258] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:35.761301 29414 ts_tablet_manager.cc:1905] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:35.761770 29413 ts_tablet_manager.cc:1905] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:35.763949 29413 ts_tablet_manager.cc:1918] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:35.764062 29413 log.cc:1196] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/1380866eeb8a4bae8913fdd61f1495b1
I20260927 00:37:35.764338 29413 ts_tablet_manager.cc:1939] T 1380866eeb8a4bae8913fdd61f1495b1 P 23a9aa3f4a5f4761a1bdbdbb43a3997e: Deleting consensus metadata
I20260927 00:37:35.765587 28908 catalog_manager.cc:4841] TS 23a9aa3f4a5f4761a1bdbdbb43a3997e (127.10.11.195:37877): tablet 1380866eeb8a4bae8913fdd61f1495b1 (table test-table [id=336a21048bf34f198582b1f9f8a48f0a]) successfully deleted
I20260927 00:37:35.765697 29414 ts_tablet_manager.cc:1918] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:35.765799 29414 log.cc:1196] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/1380866eeb8a4bae8913fdd61f1495b1
I20260927 00:37:35.766146 29414 ts_tablet_manager.cc:1939] T 1380866eeb8a4bae8913fdd61f1495b1 P 610cb503cca64a918af8dd83f47002f2: Deleting consensus metadata
W20260927 00:37:35.767233 28908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:45807 (error 108)
I20260927 00:37:35.767731 28908 catalog_manager.cc:4841] TS 610cb503cca64a918af8dd83f47002f2 (127.10.11.194:44273): tablet 1380866eeb8a4bae8913fdd61f1495b1 (table test-table [id=336a21048bf34f198582b1f9f8a48f0a]) successfully deleted
I20260927 00:37:35.768206 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29113
I20260927 00:37:35.768278 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:35.768020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:45807 (local address 127.0.0.1:41246)
0927 00:37:35.768147 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:35.768208 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:37:35.770912 28908 catalog_manager.cc:4584] TS 098ba9ed023d4c0c87e5be2bb90dc33f (127.10.11.193:45807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1380866eeb8a4bae8913fdd61f1495b1: Network error: Client connection negotiation failed: client connection to 127.10.11.193:45807: connect: Connection refused (error 111)
I20260927 00:37:35.777765 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29244
I20260927 00:37:35.785559 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 28891
2026-09-27T00:37:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1217 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:35 dist-test-slave-6gxc krb5kdc[29418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:35 dist-test-slave-6gxc krb5kdc[29418](info): set up 2 sockets
Sep 27 00:37:35 dist-test-slave-6gxc krb5kdc[29418](info): commencing operation
krb5kdc: starting...
W20260927 00:37:36.829968 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:36 dist-test-slave-6gxc krb5kdc[29418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:36.944923 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:46009
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:46009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:37.050616 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:37.050832 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:37.050869 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:37.054807 29431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:37.054852 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:37.054870 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:37.054905 29431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:37.054935 29431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:37.059957 29431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:46009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:37.060877 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:37.061803 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:37.069846 29436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.069864 29437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.069954 29431 server_base.cc:1034] running on GCE node
W20260927 00:37:37.069846 29439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.070595 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:37.071573 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:37.072755 29431 hybrid_clock.cc:648] HybridClock initialized: now 1790469457072726 us; error 45 us; skew 500 ppm
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:37.075496 29431 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:37.076412 29431 webserver.cc:460] Webserver started at http://127.10.11.254:36279/ using document root <none> and password file <none>
I20260927 00:37:37.076978 29431 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:37.077076 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:37.077318 29431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:37.079401 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "9b63c698205d45df834e1aef5986a9b0"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "837b90add1d630af50b74dddbf8d16d2"
server_key_iv: "56f938b389ee3716a670159e052ac14a"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.079993 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "9b63c698205d45df834e1aef5986a9b0"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "837b90add1d630af50b74dddbf8d16d2"
server_key_iv: "56f938b389ee3716a670159e052ac14a"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.084183 29431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:37:37.086556 29446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.087710 29431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:37.087841 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "9b63c698205d45df834e1aef5986a9b0"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "837b90add1d630af50b74dddbf8d16d2"
server_key_iv: "56f938b389ee3716a670159e052ac14a"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.087946 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:37.110761 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:37.113762 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:37.113950 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:37.120524 29431 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:46009
I20260927 00:37:37.120540 29498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:46009 every 8 connection(s)
I20260927 00:37:37.121425 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:37.124001 29499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:37.129042 29499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: Bootstrap starting.
I20260927 00:37:37.130348 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29431
I20260927 00:37:37.130496 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:37.130795 10287 external_mini_cluster.cc:1428] Setting key a951ba87fbfc1a857a9d67f795a73cf8
I20260927 00:37:37.131629 29499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:37.132354 29499 log.cc:824] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:37.134435 29499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: No bootstrap required, opened a new log
I20260927 00:37:37.138612 29499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } }
I20260927 00:37:37.138899 29499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:37.138999 29499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b63c698205d45df834e1aef5986a9b0, State: Initialized, Role: FOLLOWER
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:37.139511 29499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } }
I20260927 00:37:37.139683 29499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:37.139750 29499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:37.139849 29499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:37.140796 29499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } }
I20260927 00:37:37.141181 29499 leader_election.cc:302] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b63c698205d45df834e1aef5986a9b0; no voters:
I20260927 00:37:37.141526 29499 leader_election.cc:288] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:37.141636 29504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:37.141821 29504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 9b63c698205d45df834e1aef5986a9b0, State: Running, Role: LEADER
I20260927 00:37:37.142227 29504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } }
I20260927 00:37:37.142882 29499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:37.144178 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b63c698205d45df834e1aef5986a9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } } }
I20260927 00:37:37.144160 29506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b63c698205d45df834e1aef5986a9b0. Latest consensus state: current_term: 1 leader_uuid: "9b63c698205d45df834e1aef5986a9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63c698205d45df834e1aef5986a9b0" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 46009 } } }
I20260927 00:37:37.144318 29506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:37.144318 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:37.145947 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.131832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.10.11.254:46009)
0927 00:37:37.132257 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.132272 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.132303 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:37:37.132937 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.132952 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.133257 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.133511 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:37.134421 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.135401 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.136593 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.137046 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.137159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.140149 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:37.140154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:37.140167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:37.140193 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:37.142246 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.142754 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.142760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.142766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.142899 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.143219 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.143222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.143224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.143767 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:37.143964 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.144863 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.145181 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
W20260927 00:37:37.148020 29519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:37:37.148159 29519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:37:37.148229 29514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:37.149058 29514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:37.154511 29514 catalog_manager.cc:1347] Generated new cluster ID: 5cc377dccd2c40058cdb775a355b2901
I20260927 00:37:37.154592 29514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:37.163843 29514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:37.164637 29514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:37.176155 29514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: Generated new TSK 0
I20260927 00:37:37.176869 29514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.11.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
Entry for principal kudu/127.10.11.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab.
I20260927 00:37:37.210664 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:37.316239 29525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:37.316460 29525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:37.316500 29525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:37.320333 29525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:37.320379 29525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:37.320432 29525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:37.325378 29525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.193.keytab
--principal=kudu/127.10.11.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:37.326298 29525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:37.327172 29525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:37.336725 29530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.336719 29531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.336762 29525 server_base.cc:1034] running on GCE node
W20260927 00:37:37.336714 29533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.337369 29525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:37.337905 29525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:37.339097 29525 hybrid_clock.cc:648] HybridClock initialized: now 1790469457339095 us; error 66 us; skew 500 ppm
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:37.341670 29525 init.cc:377] Logged in from keytab as kudu/127.10.11.193@KRBTEST.COM (short username kudu)
I20260927 00:37:37.342525 29525 webserver.cc:460] Webserver started at http://127.10.11.193:45873/ using document root <none> and password file <none>
I20260927 00:37:37.343004 29525 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:37.343101 29525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:37.343307 29525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:37.345315 29525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "eca5a54143a34cee926169eedfd6b8c3"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "3d3664a376791d49d7033adfcd6489db"
server_key_iv: "9391166acc47240c0bd45c28776e36c0"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.345798 29525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "eca5a54143a34cee926169eedfd6b8c3"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "3d3664a376791d49d7033adfcd6489db"
server_key_iv: "9391166acc47240c0bd45c28776e36c0"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.349174 29525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:37:37.351348 29540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.352486 29525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:37:37.352574 29525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "eca5a54143a34cee926169eedfd6b8c3"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "3d3664a376791d49d7033adfcd6489db"
server_key_iv: "9391166acc47240c0bd45c28776e36c0"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.352648 29525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:37.365211 29525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:37.368251 29525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:37.368387 29525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:37.368835 29525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:37.369547 29525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:37.369601 29525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.369638 29525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:37.369659 29525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.378228 29525 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:34375
I20260927 00:37:37.378298 29653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:34375 every 8 connection(s)
I20260927 00:37:37.379030 29525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:37.385782 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29525
I20260927 00:37:37.385923 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:37.386281 10287 external_mini_cluster.cc:1428] Setting key 171c4e895c533763fd2910f5e74ea3f1
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.193@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:37.392632 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.380750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:52887 (local address 127.10.11.254:46009)
0927 00:37:37.380912 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.380917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.381677 (+ 760us) server_negotiation.cc:407] Connection header received
0927 00:37:37.382940 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.382951 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.382997 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.383070 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:37.384154 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.384833 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.385937 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.386290 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.386402 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.388826 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:37.388830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:37.388833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:37.388863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:37.390576 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.391058 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.391064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.391068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.391163 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.391532 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.391542 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.391544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.391789 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:37.391894 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.392357 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.392476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:37:37.393481 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.381089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46009 (local address 127.10.11.193:52887)
0927 00:37:37.381507 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.381547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.382738 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.383223 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.383229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.383536 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:37.384007 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.384019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.384940 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.384943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.385797 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.385806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.386435 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.386439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.386623 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.387185 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:37.387200 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:37.388669 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:37.390688 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.390692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.390702 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.390937 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.391283 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.391286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.391288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.391417 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.391891 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:37.391896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:37.392115 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.392294 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.392643 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260927 00:37:37.394800 29654 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46009
I20260927 00:37:37.395040 29654 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.11.194@KRBTEST.COM; defaulting to no policy
I20260927 00:37:37.395512 29654 heartbeater.cc:511] Master 127.10.11.254:46009 requested a full tablet report, sending...
I20260927 00:37:37.396914 29463 ts_manager.cc:194] Registered new tserver with Master: eca5a54143a34cee926169eedfd6b8c3 (127.10.11.193:34375)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.194@KRBTEST.COM" created.
I20260927 00:37:37.398198 29463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.193@KRBTEST.COM'} at 127.10.11.193:52887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
Entry for principal kudu/127.10.11.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab.
I20260927 00:37:37.411420 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:37.517357 29659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:37.517577 29659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:37.517652 29659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:37.521546 29659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:37.521600 29659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:37.521656 29659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:37.526738 29659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.194.keytab
--principal=kudu/127.10.11.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:37.527630 29659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:37.528571 29659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:37.537705 29664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.537724 29665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.537767 29667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.537935 29659 server_base.cc:1034] running on GCE node
I20260927 00:37:37.538426 29659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:37.538995 29659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:37.540177 29659 hybrid_clock.cc:648] HybridClock initialized: now 1790469457540172 us; error 30 us; skew 500 ppm
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:37.542899 29659 init.cc:377] Logged in from keytab as kudu/127.10.11.194@KRBTEST.COM (short username kudu)
I20260927 00:37:37.543812 29659 webserver.cc:460] Webserver started at http://127.10.11.194:35605/ using document root <none> and password file <none>
I20260927 00:37:37.544330 29659 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:37.544417 29659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:37.544674 29659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:37.546670 29659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "df6a3603c5fd4d4c92a6fb916b737e37"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "faf1c72ceff448094adb5a120d1093d0"
server_key_iv: "810bf832406c42c72ed09d4c71c147e2"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.547212 29659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "df6a3603c5fd4d4c92a6fb916b737e37"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "faf1c72ceff448094adb5a120d1093d0"
server_key_iv: "810bf832406c42c72ed09d4c71c147e2"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.550616 29659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:37.552870 29674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.553975 29659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:37.554143 29659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "df6a3603c5fd4d4c92a6fb916b737e37"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "faf1c72ceff448094adb5a120d1093d0"
server_key_iv: "810bf832406c42c72ed09d4c71c147e2"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.554258 29659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:37.577562 29659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:37.580735 29659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:37.580938 29659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:37.581501 29659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:37.582368 29659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:37.582436 29659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.582506 29659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:37.582551 29659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.591092 29659 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:37921
I20260927 00:37:37.591122 29787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:37921 every 8 connection(s)
I20260927 00:37:37.592013 29659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:37.597132 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29659
I20260927 00:37:37.597287 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:37:37.597594 10287 external_mini_cluster.cc:1428] Setting key d0dbed06c5de622360f17038273ab9fa
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.194@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:37.605748 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.593688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:38371 (local address 127.10.11.254:46009)
0927 00:37:37.593829 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.593833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.594723 (+ 890us) server_negotiation.cc:407] Connection header received
0927 00:37:37.595944 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.595946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.595988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.596059 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:37.597257 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.597882 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.599048 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.599271 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.599379 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.602065 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:37.602068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:37.602069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:37.602091 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:37.603523 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.604188 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.604190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.604192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.604263 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.604667 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.604669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.604671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.604829 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:37.604901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.605524 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.605631 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:37:37.606913 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.594093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46009 (local address 127.10.11.194:38371)
0927 00:37:37.594576 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.594622 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.595769 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.596214 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.596220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.596525 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:37.597095 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.597116 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.598006 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.598014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.598919 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.598929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.599417 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.599421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.599582 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.600135 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:37.600151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:37.601904 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:37.603657 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.603665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.603682 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.604076 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.604361 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.604366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.604370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.604570 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.604940 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:37.604948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:37.605268 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.605486 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.605844 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.10.11.195@KRBTEST.COM; defaulting to no policy
I20260927 00:37:37.608091 29788 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46009
I20260927 00:37:37.608351 29788 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:37.608819 29788 heartbeater.cc:511] Master 127.10.11.254:46009 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.195@KRBTEST.COM" created.
I20260927 00:37:37.609822 29463 ts_manager.cc:194] Registered new tserver with Master: df6a3603c5fd4d4c92a6fb916b737e37 (127.10.11.194:37921)
I20260927 00:37:37.610455 29463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.194@KRBTEST.COM'} at 127.10.11.194:38371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
Entry for principal kudu/127.10.11.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab.
I20260927 00:37:37.625449 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:37.731671 29793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:37.731940 29793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:37.732012 29793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:37.736124 29793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:37.736207 29793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:37.736320 29793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:37.741626 29793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.195.keytab
--principal=kudu/127.10.11.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:37.742619 29793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:37.743607 29793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:37.752998 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.753000 29801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:37.753006 29799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:37.753341 29793 server_base.cc:1034] running on GCE node
I20260927 00:37:37.753703 29793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:37.754341 29793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:37.755568 29793 hybrid_clock.cc:648] HybridClock initialized: now 1790469457755552 us; error 29 us; skew 500 ppm
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:37.758272 29793 init.cc:377] Logged in from keytab as kudu/127.10.11.195@KRBTEST.COM (short username kudu)
I20260927 00:37:37.759093 29793 webserver.cc:460] Webserver started at http://127.10.11.195:46863/ using document root <none> and password file <none>
I20260927 00:37:37.759603 29793 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:37.759697 29793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:37.759927 29793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:37.761767 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "003192e09fec48118a34d53f9c8ccf56"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "c37eb049e73b333473d53574c49f20a1"
server_key_iv: "386ab9f529d3cc15e9da18114f0eeb01"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.762321 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "003192e09fec48118a34d53f9c8ccf56"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "c37eb049e73b333473d53574c49f20a1"
server_key_iv: "386ab9f529d3cc15e9da18114f0eeb01"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.765614 29793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:37:37.767918 29808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.769034 29793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:37.769162 29793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "003192e09fec48118a34d53f9c8ccf56"
format_stamp: "Formatted at 2026-09-27 00:37:37 on dist-test-slave-6gxc"
server_key: "c37eb049e73b333473d53574c49f20a1"
server_key_iv: "386ab9f529d3cc15e9da18114f0eeb01"
server_key_version: "encryptionkey@0"
I20260927 00:37:37.769263 29793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:37.791967 29793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:37.794636 29793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:37.794823 29793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:37.795351 29793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:37.796114 29793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:37.796182 29793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.796247 29793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:37.796299 29793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:37.804803 29793 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:42627
I20260927 00:37:37.804843 29921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:42627 every 8 connection(s)
I20260927 00:37:37.805600 29793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:37.810721 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29793
I20260927 00:37:37.810889 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:37.811218 10287 external_mini_cluster.cc:1428] Setting key e9549a63cd11191e59ff1f5eeeb50a8b
Sep 27 00:37:37 dist-test-slave-6gxc krb5kdc[29418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.195@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:37.819041 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.807240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:53383 (local address 127.10.11.254:46009)
0927 00:37:37.807376 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.807379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.808158 (+ 779us) server_negotiation.cc:407] Connection header received
0927 00:37:37.809429 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.809433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.809527 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.809625 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:37.810810 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.811565 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.812977 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.813257 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.813340 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.815650 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:37.815653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:37.815655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:37.815677 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:37.817130 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.817561 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.817563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.817564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.817634 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.818071 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.818073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.818074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.818319 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:37.818430 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.818837 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.818924 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:37:37.819895 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.807581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:46009 (local address 127.10.11.195:53383)
0927 00:37:37.808002 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.808041 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.809221 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.809791 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.809797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.810204 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:37.810652 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.810665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.811738 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.811742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.812851 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.812861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.813396 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.813399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.813540 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.814006 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:37:37.814021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:37.815492 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:37.817231 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.817235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.817244 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.817478 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.817809 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:37:37.817812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:37:37.817813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:37:37.817971 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:37:37.818435 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:37.818439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:37.818639 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.818789 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.819010 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260927 00:37:37.820894 29922 heartbeater.cc:347] Connected to a master server at 127.10.11.254:46009
I20260927 00:37:37.821092 29922 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:37.821480 29922 heartbeater.cc:511] Master 127.10.11.254:46009 requested a full tablet report, sending...
I20260927 00:37:37.822386 29463 ts_manager.cc:194] Registered new tserver with Master: 003192e09fec48118a34d53f9c8ccf56 (127.10.11.195:42627)
I20260927 00:37:37.822870 29463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.11.195@KRBTEST.COM'} at 127.10.11.195:53383
I20260927 00:37:37.824605 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:37.833195 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.826440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.10.11.254:46009)
0927 00:37:37.826565 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.826569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.826761 (+ 192us) server_negotiation.cc:407] Connection header received
0927 00:37:37.826845 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.826847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.826881 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.826951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:37.827533 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.828001 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.828786 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.828986 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.829065 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.830330 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:37.830333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:37.830335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:37.830355 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:37.831732 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.832122 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.832124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.832125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.832198 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:37.832451 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:37.832453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:37.832454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:37.832646 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:37.832740 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.832902 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.833000 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:37:37.835814 29463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:37.838579 29463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:37.852291 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.846616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:37921 (local address 127.0.0.1:34400)
0927 00:37:37.847419 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.847432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.847557 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.848141 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.848146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.848169 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:37.848282 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.848290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.849779 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.849782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.850856 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.850864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.851869 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.851872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.851946 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.851973 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.852070 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.852148 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":56,"thread_start_us":96,"threads_started":1}
I20260927 00:37:37.852362 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.847067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.10.11.194:37921)
0927 00:37:37.847360 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.847371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.847660 (+ 289us) server_negotiation.cc:407] Connection header received
0927 00:37:37.847745 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.847751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.847961 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.848135 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:37.848406 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.849627 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.851016 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.851748 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.851884 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.852023 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.852125 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.852231 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260927 00:37:37.852862 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.846969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34375 (local address 127.0.0.1:45532)
0927 00:37:37.847717 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.847728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.847796 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.848329 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.848332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.848371 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:37.848560 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.848568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.850049 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.850052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.851234 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.851239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.852266 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.852270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.852552 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.852580 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.852674 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.852744 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":85,"threads_started":1}
I20260927 00:37:37.853673 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.847348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.10.11.193:34375)
0927 00:37:37.847756 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.847767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.847788 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:37:37.847884 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.847889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.848043 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.848182 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:37.848701 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.849931 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.851350 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.852150 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.852249 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.852370 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.853412 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.853507 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
I20260927 00:37:37.853832 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.846346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42627 (local address 127.0.0.1:56696)
0927 00:37:37.846832 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:37.846877 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:37.846986 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:37.848424 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:37.848426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:37.848443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:37.848532 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.848541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.849479 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.849482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.851225 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:37.851236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.853390 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.853393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:37.853511 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.853533 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:37:37.853613 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:37:37.853680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":343,"thread_start_us":66,"threads_started":1}
I20260927 00:37:37.853855 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:37.846597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.10.11.195:42627)
0927 00:37:37.847991 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:37.847999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:37.848015 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:37:37.848098 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:37.848103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:37.848278 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:37.848443 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:37.848656 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.849349 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.852430 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:37.853266 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:37.853376 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:37.853497 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:37.853651 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:37:37.853732 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":122,"threads_started":1}
I20260927 00:37:37.855630 29722 tablet_service.cc:1467] Processing CreateTablet for tablet 60e8096f9a23433296185e7e861cbfb1 (DEFAULT_TABLE table=test-table [id=e1c2960097d648a8b9b018f17ad928a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:37.856716 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e8096f9a23433296185e7e861cbfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:37.856397 29856 tablet_service.cc:1467] Processing CreateTablet for tablet 60e8096f9a23433296185e7e861cbfb1 (DEFAULT_TABLE table=test-table [id=e1c2960097d648a8b9b018f17ad928a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:37.856372 29588 tablet_service.cc:1467] Processing CreateTablet for tablet 60e8096f9a23433296185e7e861cbfb1 (DEFAULT_TABLE table=test-table [id=e1c2960097d648a8b9b018f17ad928a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:37.857266 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e8096f9a23433296185e7e861cbfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:37.857323 29588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e8096f9a23433296185e7e861cbfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:37.863125 29939 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Bootstrap starting.
I20260927 00:37:37.864459 29941 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Bootstrap starting.
I20260927 00:37:37.864459 29940 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Bootstrap starting.
I20260927 00:37:37.865255 29939 tablet_bootstrap.cc:654] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:37.866554 29941 tablet_bootstrap.cc:654] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:37.866554 29940 tablet_bootstrap.cc:654] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:37.866647 29939 log.cc:824] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:37.867794 29941 log.cc:824] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:37.867794 29940 log.cc:824] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:37.868603 29939 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: No bootstrap required, opened a new log
I20260927 00:37:37.868803 29939 ts_tablet_manager.cc:1397] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:37:37.869519 29941 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: No bootstrap required, opened a new log
I20260927 00:37:37.869721 29941 ts_tablet_manager.cc:1397] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:37:37.869812 29940 tablet_bootstrap.cc:492] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: No bootstrap required, opened a new log
I20260927 00:37:37.869987 29940 ts_tablet_manager.cc:1397] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:37:37.872491 29941 raft_consensus.cc:348] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.872725 29941 raft_consensus.cc:374] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:37.872795 29941 raft_consensus.cc:729] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a3603c5fd4d4c92a6fb916b737e37, State: Initialized, Role: FOLLOWER
I20260927 00:37:37.872632 29939 raft_consensus.cc:348] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.872857 29939 raft_consensus.cc:374] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:37.872949 29939 raft_consensus.cc:729] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003192e09fec48118a34d53f9c8ccf56, State: Initialized, Role: FOLLOWER
I20260927 00:37:37.873250 29941 consensus_queue.cc:260] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.873390 29939 consensus_queue.cc:260] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.873677 29940 raft_consensus.cc:348] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.873911 29940 raft_consensus.cc:374] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:37.873931 29788 heartbeater.cc:503] Master 127.10.11.254:46009 was elected leader, sending a full tablet report...
I20260927 00:37:37.873956 29940 raft_consensus.cc:729] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca5a54143a34cee926169eedfd6b8c3, State: Initialized, Role: FOLLOWER
I20260927 00:37:37.874253 29941 ts_tablet_manager.cc:1428] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 00:37:37.874380 29939 ts_tablet_manager.cc:1428] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:37:37.874346 29940 consensus_queue.cc:260] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:37.874676 29922 heartbeater.cc:503] Master 127.10.11.254:46009 was elected leader, sending a full tablet report...
I20260927 00:37:37.875173 29940 ts_tablet_manager.cc:1428] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:37:37.875517 29654 heartbeater.cc:503] Master 127.10.11.254:46009 was elected leader, sending a full tablet report...
W20260927 00:37:37.880103 29655 tablet.cc:2369] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:38.007673 29947 raft_consensus.cc:482] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:38.007858 29947 raft_consensus.cc:504] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:38.008932 29947 leader_election.cc:288] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 003192e09fec48118a34d53f9c8ccf56 (127.10.11.195:42627), df6a3603c5fd4d4c92a6fb916b737e37 (127.10.11.194:37921)
I20260927 00:37:38.013031 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.009290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:42443 (local address 127.10.11.195:42627)
0927 00:37:38.009432 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.009437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.009696 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:37:38.009826 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.009828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.009895 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.009975 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:38.010314 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.010837 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012011 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.012429 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012556 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.012617 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.012839 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.012909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:37:38.013057 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.009290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:50671 (local address 127.10.11.194:37921)
0927 00:37:38.009494 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.009499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.009602 (+ 103us) server_negotiation.cc:407] Connection header received
0927 00:37:38.009763 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.009769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.009834 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.009891 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:38.010251 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.010763 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012056 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.012431 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012555 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.012613 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.012884 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.012953 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 00:37:38.013042 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.009174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:42627 (local address 127.10.11.193:42443)
0927 00:37:38.009543 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.009599 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.009723 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.010008 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.010013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.010028 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:38.010185 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.010208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.010953 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.010956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.011886 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.011895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.012552 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.012707 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.012735 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.012795 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.012879 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":55,"thread_start_us":93,"threads_started":1}
I20260927 00:37:38.013053 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.009174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:37921 (local address 127.10.11.193:50671)
0927 00:37:38.009497 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.009510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.009619 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.009940 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.009947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.009967 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:38.010133 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.010140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.010879 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.010884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.011906 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.011925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.012552 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.012556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.012717 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.012756 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.012850 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.012919 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":98,"thread_start_us":105,"threads_started":1}
I20260927 00:37:38.013659 29876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e8096f9a23433296185e7e861cbfb1" candidate_uuid: "eca5a54143a34cee926169eedfd6b8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003192e09fec48118a34d53f9c8ccf56" is_pre_election: true
I20260927 00:37:38.013880 29742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e8096f9a23433296185e7e861cbfb1" candidate_uuid: "eca5a54143a34cee926169eedfd6b8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" is_pre_election: true
I20260927 00:37:38.013957 29876 raft_consensus.cc:2454] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca5a54143a34cee926169eedfd6b8c3 in term 0.
I20260927 00:37:38.014168 29742 raft_consensus.cc:2454] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca5a54143a34cee926169eedfd6b8c3 in term 0.
I20260927 00:37:38.014477 29543 leader_election.cc:302] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003192e09fec48118a34d53f9c8ccf56, eca5a54143a34cee926169eedfd6b8c3; no voters:
I20260927 00:37:38.014753 29947 raft_consensus.cc:2781] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:38.014834 29947 raft_consensus.cc:482] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:38.014876 29947 raft_consensus.cc:3037] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:38.015875 29947 raft_consensus.cc:504] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:38.016216 29947 leader_election.cc:288] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [CANDIDATE]: Term 1 election: Requested vote from peers 003192e09fec48118a34d53f9c8ccf56 (127.10.11.195:42627), df6a3603c5fd4d4c92a6fb916b737e37 (127.10.11.194:37921)
I20260927 00:37:38.016544 29876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e8096f9a23433296185e7e861cbfb1" candidate_uuid: "eca5a54143a34cee926169eedfd6b8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003192e09fec48118a34d53f9c8ccf56"
I20260927 00:37:38.016619 29742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e8096f9a23433296185e7e861cbfb1" candidate_uuid: "eca5a54143a34cee926169eedfd6b8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a3603c5fd4d4c92a6fb916b737e37"
I20260927 00:37:38.016657 29876 raft_consensus.cc:3037] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:38.016767 29742 raft_consensus.cc:3037] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:38.017684 29876 raft_consensus.cc:2454] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca5a54143a34cee926169eedfd6b8c3 in term 1.
I20260927 00:37:38.017854 29742 raft_consensus.cc:2454] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca5a54143a34cee926169eedfd6b8c3 in term 1.
I20260927 00:37:38.017988 29543 leader_election.cc:302] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003192e09fec48118a34d53f9c8ccf56, eca5a54143a34cee926169eedfd6b8c3; no voters:
I20260927 00:37:38.018184 29947 raft_consensus.cc:2781] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:38.018440 29947 raft_consensus.cc:686] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 1 LEADER]: Becoming Leader. State: Replica: eca5a54143a34cee926169eedfd6b8c3, State: Running, Role: LEADER
I20260927 00:37:38.018761 29947 consensus_queue.cc:237] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } }
I20260927 00:37:38.021795 29461 catalog_manager.cc:5495] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eca5a54143a34cee926169eedfd6b8c3 (127.10.11.193). New cstate: current_term: 1 leader_uuid: "eca5a54143a34cee926169eedfd6b8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca5a54143a34cee926169eedfd6b8c3" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 34375 } health_report { overall_health: HEALTHY } } }
I20260927 00:37:38.037663 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.034199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.10.11.193:34375)
0927 00:37:38.034324 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.034329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.034373 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:37:38.034566 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.034569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.034621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.034732 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:38.034974 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.035465 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.036305 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.036520 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.036629 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.036925 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:38.037292 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:38.037419 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.037477 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.037527 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:37:38.043138 29742 raft_consensus.cc:1261] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 1 FOLLOWER]: Refusing update from remote peer eca5a54143a34cee926169eedfd6b8c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:38.043147 29876 raft_consensus.cc:1261] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 1 FOLLOWER]: Refusing update from remote peer eca5a54143a34cee926169eedfd6b8c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:38.043850 29950 consensus_queue.cc:1035] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003192e09fec48118a34d53f9c8ccf56" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:38.044049 29947 consensus_queue.cc:1035] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a3603c5fd4d4c92a6fb916b737e37" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:38.051513 29954 mvcc.cc:205] Tried to move back new op lower bound from 7333762900137496576 to 7333762900047564800. Current Snapshot: MvccSnapshot[applied={T|T < 7333762900137496576}]
I20260927 00:37:38.052642 29955 mvcc.cc:205] Tried to move back new op lower bound from 7333762900137496576 to 7333762900047564800. Current Snapshot: MvccSnapshot[applied={T|T < 7333762900137496576}]
I20260927 00:37:38.060665 29463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:38.060834 29463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:38.063054 29463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b63c698205d45df834e1aef5986a9b0: Sending DeleteTablet for 3 replicas of tablet 60e8096f9a23433296185e7e861cbfb1
I20260927 00:37:38.063794 29722 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e8096f9a23433296185e7e861cbfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:38 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:34400
I20260927 00:37:38.063838 29856 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e8096f9a23433296185e7e861cbfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:38 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:56696
I20260927 00:37:38.064134 29588 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e8096f9a23433296185e7e861cbfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:38 UTC) from {username='kudu', principal='kudu/127.10.11.254@KRBTEST.COM'} at 127.0.0.1:45532
I20260927 00:37:38.064213 29962 tablet_replica.cc:330] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: stopping tablet replica
I20260927 00:37:38.064464 29962 raft_consensus.cc:2229] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:38.064570 29964 tablet_replica.cc:330] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3: stopping tablet replica
I20260927 00:37:38.064815 29964 raft_consensus.cc:2229] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:37:38.064973 29962 raft_consensus.cc:2258] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:38.065145 29964 raft_consensus.cc:2258] T 60e8096f9a23433296185e7e861cbfb1 P eca5a54143a34cee926169eedfd6b8c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:38.065289 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29525
I20260927 00:37:38.067267 29962 ts_tablet_manager.cc:1905] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:38.069319 29962 ts_tablet_manager.cc:1918] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:38.069429 29962 log.cc:1196] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/60e8096f9a23433296185e7e861cbfb1
I20260927 00:37:38.069657 29963 tablet_replica.cc:330] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: stopping tablet replica
I20260927 00:37:38.069766 29962 ts_tablet_manager.cc:1939] T 60e8096f9a23433296185e7e861cbfb1 P df6a3603c5fd4d4c92a6fb916b737e37: Deleting consensus metadata
I20260927 00:37:38.069944 29963 raft_consensus.cc:2229] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:38.070236 29963 raft_consensus.cc:2258] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:38.070935 29447 catalog_manager.cc:4841] TS df6a3603c5fd4d4c92a6fb916b737e37 (127.10.11.194:37921): tablet 60e8096f9a23433296185e7e861cbfb1 (table test-table [id=e1c2960097d648a8b9b018f17ad928a4]) successfully deleted
I20260927 00:37:38.071473 29963 ts_tablet_manager.cc:1905] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:37:38.071784 29449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:34375 (error 108)
I20260927 00:37:38.072484 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29659
I20260927 00:37:38.073122 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.072776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:34375 (local address 127.0.0.1:45552)
0927 00:37:38.072950 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.073019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 00:37:38.073273 29449 catalog_manager.cc:4584] TS eca5a54143a34cee926169eedfd6b8c3 (127.10.11.193:34375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60e8096f9a23433296185e7e861cbfb1: Network error: Client connection negotiation failed: client connection to 127.10.11.193:34375: connect: Connection refused (error 111)
I20260927 00:37:38.073805 29963 ts_tablet_manager.cc:1918] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:38.073911 29963 log.cc:1196] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/60e8096f9a23433296185e7e861cbfb1
I20260927 00:37:38.074285 29963 ts_tablet_manager.cc:1939] T 60e8096f9a23433296185e7e861cbfb1 P 003192e09fec48118a34d53f9c8ccf56: Deleting consensus metadata
I20260927 00:37:38.075227 29449 catalog_manager.cc:4841] TS 003192e09fec48118a34d53f9c8ccf56 (127.10.11.195:42627): tablet 60e8096f9a23433296185e7e861cbfb1 (table test-table [id=e1c2960097d648a8b9b018f17ad928a4]) successfully deleted
I20260927 00:37:38.078764 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29793
I20260927 00:37:38.084326 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29431
2026-09-27T00:37:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2299 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:38Z Disabled control of system clock
I20260927 00:37:38.113380 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43027
--webserver_interface=127.10.11.254
--webserver_port=0
--builtin_ntp_servers=127.10.11.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:43027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:38.216984 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:38.217203 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:38.217236 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:38.221220 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:38.221268 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:38.221284 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:38.221300 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:38.221315 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:38.226227 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:43027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:38.227128 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:38.228366 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:38.235963 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.236053 29969 server_base.cc:1034] running on GCE node
W20260927 00:37:38.235972 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:38.235978 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.236713 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:38.237674 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:38.238824 29969 hybrid_clock.cc:648] HybridClock initialized: now 1790469458238796 us; error 41 us; skew 500 ppm
I20260927 00:37:38.240585 29969 webserver.cc:460] Webserver started at http://127.10.11.254:43611/ using document root <none> and password file <none>
I20260927 00:37:38.241097 29969 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:38.241154 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:38.241376 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:38.243288 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "bb81a609461d42fda2016a2028379f07"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "119112216d15b8cec1e8ce0a37a0ca24"
server_key_iv: "c4effe26ed4f6fb8c3693fcfe80dcdfc"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.243814 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "bb81a609461d42fda2016a2028379f07"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "119112216d15b8cec1e8ce0a37a0ca24"
server_key_iv: "c4effe26ed4f6fb8c3693fcfe80dcdfc"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.247213 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:38.249394 29983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.250566 29969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:38.250684 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "bb81a609461d42fda2016a2028379f07"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "119112216d15b8cec1e8ce0a37a0ca24"
server_key_iv: "c4effe26ed4f6fb8c3693fcfe80dcdfc"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.250775 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:38.275081 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:38.275697 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:38.275882 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:38.282244 29969 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:43027
I20260927 00:37:38.282262 30035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:43027 every 8 connection(s)
I20260927 00:37:38.283125 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:38.285883 30036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:38.288368 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 29969
I20260927 00:37:38.288487 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:38.288818 10287 external_mini_cluster.cc:1428] Setting key 3bbb380b473f92e4ebc2e4201d8ae00e
I20260927 00:37:38.291677 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: Bootstrap starting.
I20260927 00:37:38.293635 30036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:38.294361 30036 log.cc:824] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:38.296036 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: No bootstrap required, opened a new log
I20260927 00:37:38.297514 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.290154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.10.11.254:43027)
0927 00:37:38.290760 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.290774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.290806 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:37:38.291394 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.291409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.291767 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.292107 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:38.292510 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.293419 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.294361 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.295023 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.295182 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.296007 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:38.296013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:38.296032 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:38.296415 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:38.296465 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:38.296484 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:38.296582 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.296691 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.296925 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":146,"threads_started":1}
I20260927 00:37:38.299278 30036 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } }
I20260927 00:37:38.299512 30036 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:38.299566 30036 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb81a609461d42fda2016a2028379f07, State: Initialized, Role: FOLLOWER
I20260927 00:37:38.300009 30036 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } }
I20260927 00:37:38.300158 30036 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:38.300204 30036 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:38.300271 30036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:38.300987 30036 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } }
I20260927 00:37:38.301299 30036 leader_election.cc:302] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb81a609461d42fda2016a2028379f07; no voters:
I20260927 00:37:38.301518 30036 leader_election.cc:288] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:38.301635 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:38.301858 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [term 1 LEADER]: Becoming Leader. State: Replica: bb81a609461d42fda2016a2028379f07, State: Running, Role: LEADER
I20260927 00:37:38.302165 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } }
I20260927 00:37:38.302737 30036 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:38.303791 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb81a609461d42fda2016a2028379f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } } }
I20260927 00:37:38.303814 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb81a609461d42fda2016a2028379f07. Latest consensus state: current_term: 1 leader_uuid: "bb81a609461d42fda2016a2028379f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb81a609461d42fda2016a2028379f07" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43027 } } }
I20260927 00:37:38.303923 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:38.303992 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:38.304276 30050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:38.307099 30050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:38.311900 30050 catalog_manager.cc:1347] Generated new cluster ID: f0d45c4e4e224959a97a84c5e0ee7313
I20260927 00:37:38.311954 30050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:38.323179 30050 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:38.324267 30050 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:38.333130 30050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: Generated new TSK 0
I20260927 00:37:38.333803 30050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:38.342029 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.193:0
--local_ip_for_outbound_sockets=127.10.11.193
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--builtin_ntp_servers=127.10.11.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:37:38.448601 30060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:38.448848 30060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:38.448920 30060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:38.452836 30060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:38.452905 30060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:38.452996 30060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.193
I20260927 00:37:38.457657 30060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.11.193
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.193
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:38.458613 30060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:38.459877 30060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:38.468982 30066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:38.468940 30068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.468991 30060 server_base.cc:1034] running on GCE node
W20260927 00:37:38.468940 30065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.469555 30060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:38.470171 30060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:38.471326 30060 hybrid_clock.cc:648] HybridClock initialized: now 1790469458471307 us; error 30 us; skew 500 ppm
I20260927 00:37:38.472944 30060 webserver.cc:460] Webserver started at http://127.10.11.193:44623/ using document root <none> and password file <none>
I20260927 00:37:38.473465 30060 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:38.473528 30060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:38.473762 30060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:38.475677 30060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data/instance:
uuid: "ecb44e086f7d409b9cea437db210d0d4"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "3736ee0acafeb8b5b3411f4ed2463b46"
server_key_iv: "caaa0058a01f5d677afe3d4aa70c7f13"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.476202 30060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance:
uuid: "ecb44e086f7d409b9cea437db210d0d4"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "3736ee0acafeb8b5b3411f4ed2463b46"
server_key_iv: "caaa0058a01f5d677afe3d4aa70c7f13"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.479615 30060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:37:38.481801 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.482957 30060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:38.483088 30060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal
uuid: "ecb44e086f7d409b9cea437db210d0d4"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "3736ee0acafeb8b5b3411f4ed2463b46"
server_key_iv: "caaa0058a01f5d677afe3d4aa70c7f13"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.483189 30060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:38.495059 30060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:38.495623 30060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:38.495798 30060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:38.496304 30060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:38.497138 30060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:38.497208 30060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.497275 30060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:38.497318 30060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.505580 30060 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.193:46083
I20260927 00:37:38.505604 30187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.193:46083 every 8 connection(s)
I20260927 00:37:38.506474 30060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/data/info.pb
I20260927 00:37:38.507786 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30060
I20260927 00:37:38.507936 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-0/wal/instance
I20260927 00:37:38.508307 10287 external_mini_cluster.cc:1428] Setting key 1d1cc420e0d4929f996b3564f86c116c
I20260927 00:37:38.510284 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.194:0
--local_ip_for_outbound_sockets=127.10.11.194
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--builtin_ntp_servers=127.10.11.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:38.516371 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.508354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.193:34169 (local address 127.10.11.254:43027)
0927 00:37:38.508510 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.508515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.509629 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 00:37:38.511152 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.511158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.511220 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.511316 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:38.512518 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.513046 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.514214 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.514651 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.514785 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.515795 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:38.515801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:38.515806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:38.515824 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:38.515852 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:38.515865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:38.515948 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.516099 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.516169 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:37:38.517027 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.508785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43027 (local address 127.10.11.193:34169)
0927 00:37:38.509486 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.509535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.510945 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.511532 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.511543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.511653 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:38.512347 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.512370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.513156 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.513158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.514041 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.514050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.514794 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.514796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.514926 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.515553 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:38.515554 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:38.515588 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:38.515644 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:38.515961 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:38.515967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:38.515971 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.516063 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.516257 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260927 00:37:38.518289 30188 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43027
I20260927 00:37:38.518510 30188 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:38.518966 30188 heartbeater.cc:511] Master 127.10.11.254:43027 requested a full tablet report, sending...
I20260927 00:37:38.520551 30000 ts_manager.cc:194] Registered new tserver with Master: ecb44e086f7d409b9cea437db210d0d4 (127.10.11.193:46083)
I20260927 00:37:38.522202 30000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.193:34169
W20260927 00:37:38.618340 30191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:38.618561 30191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:38.618595 30191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:38.622471 30191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:38.622514 30191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:38.622563 30191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.194
I20260927 00:37:38.627013 30191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.11.194
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.194
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:38.627862 30191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:38.629109 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:38.638197 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:38.638206 30196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.638279 30191 server_base.cc:1034] running on GCE node
W20260927 00:37:38.638356 30199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.638813 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:38.639372 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:38.640569 30191 hybrid_clock.cc:648] HybridClock initialized: now 1790469458640547 us; error 31 us; skew 500 ppm
I20260927 00:37:38.642372 30191 webserver.cc:460] Webserver started at http://127.10.11.194:43525/ using document root <none> and password file <none>
I20260927 00:37:38.642925 30191 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:38.643010 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:38.643221 30191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:38.645090 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data/instance:
uuid: "499b93e970bf4883aebde93ff8376954"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "fcecdcc26d9817f9491dc344356374cd"
server_key_iv: "54f41b55bf6202a69eb6c475ef8fb5f8"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.645618 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance:
uuid: "499b93e970bf4883aebde93ff8376954"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "fcecdcc26d9817f9491dc344356374cd"
server_key_iv: "54f41b55bf6202a69eb6c475ef8fb5f8"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.649082 30191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:37:38.651533 30205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.652627 30191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:37:38.652758 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal
uuid: "499b93e970bf4883aebde93ff8376954"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "fcecdcc26d9817f9491dc344356374cd"
server_key_iv: "54f41b55bf6202a69eb6c475ef8fb5f8"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.652872 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:38.671391 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:38.672026 30191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:38.672206 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:38.672745 30191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:38.673681 30191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:38.673750 30191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.673823 30191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:38.673866 30191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.682957 30191 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.194:35653
I20260927 00:37:38.683002 30318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.194:35653 every 8 connection(s)
I20260927 00:37:38.683808 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/data/info.pb
I20260927 00:37:38.686663 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30191
I20260927 00:37:38.686817 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal/instance
I20260927 00:37:38.687187 10287 external_mini_cluster.cc:1428] Setting key d6c6f6e847b23dd36337e96e1f495ee7
I20260927 00:37:38.689126 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.11.195:0
--local_ip_for_outbound_sockets=127.10.11.195
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--builtin_ntp_servers=127.10.11.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:37:38.694231 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.685542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.194:35551 (local address 127.10.11.254:43027)
0927 00:37:38.685688 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.685692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.686875 (+ 1183us) server_negotiation.cc:407] Connection header received
0927 00:37:38.688420 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.688423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.688478 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.688552 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:38.689516 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.690407 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.692147 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.692355 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.692434 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.693714 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:38.693717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:38.693719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:38.693730 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:38.693748 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:38.693755 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:38.693814 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.694015 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.694063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:37:38.695415 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.685897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43027 (local address 127.10.11.194:35551)
0927 00:37:38.686692 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.686757 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.688230 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.688720 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.688727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.688806 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:38.689374 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.689389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.690554 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.690561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.692017 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.692031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.692508 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.692515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.692742 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.693402 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:38.693406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:38.693470 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:38.693554 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:38.693845 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:38.693858 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:38.693866 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.694005 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.694344 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":162,"threads_started":1}
I20260927 00:37:38.696599 30319 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43027
I20260927 00:37:38.696826 30319 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:38.697289 30319 heartbeater.cc:511] Master 127.10.11.254:43027 requested a full tablet report, sending...
I20260927 00:37:38.698328 30000 ts_manager.cc:194] Registered new tserver with Master: 499b93e970bf4883aebde93ff8376954 (127.10.11.194:35653)
I20260927 00:37:38.699184 30000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.194:35551
W20260927 00:37:38.797290 30322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:38.797508 30322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:38.797544 30322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:38.801327 30322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:38.801367 30322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:38.801417 30322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.11.195
I20260927 00:37:38.806200 30322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.11.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.11.195
--webserver_port=0
--tserver_master_addrs=127.10.11.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.11.195
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:38.807021 30322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:38.808313 30322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:38.817277 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:38.817265 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:38.817262 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:38.817327 30322 server_base.cc:1034] running on GCE node
I20260927 00:37:38.817852 30322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:38.818485 30322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:38.819669 30322 hybrid_clock.cc:648] HybridClock initialized: now 1790469458819656 us; error 24 us; skew 500 ppm
I20260927 00:37:38.825829 30322 webserver.cc:460] Webserver started at http://127.10.11.195:40615/ using document root <none> and password file <none>
I20260927 00:37:38.826422 30322 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:38.826504 30322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:38.826718 30322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:38.828648 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data/instance:
uuid: "04a0bd8e9ff54a9b929614d1a12b8235"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "faec0326133c34a30ee471feda0d272f"
server_key_iv: "1dcc3515ba812225c0b9de08583a4899"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.829178 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance:
uuid: "04a0bd8e9ff54a9b929614d1a12b8235"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "faec0326133c34a30ee471feda0d272f"
server_key_iv: "1dcc3515ba812225c0b9de08583a4899"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.832604 30322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 00:37:38.835141 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.836385 30322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:37:38.836525 30322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal
uuid: "04a0bd8e9ff54a9b929614d1a12b8235"
format_stamp: "Formatted at 2026-09-27 00:37:38 on dist-test-slave-6gxc"
server_key: "faec0326133c34a30ee471feda0d272f"
server_key_iv: "1dcc3515ba812225c0b9de08583a4899"
server_key_version: "encryptionkey@0"
I20260927 00:37:38.836634 30322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:38.881114 30322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:38.881800 30322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:38.882000 30322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:38.882609 30322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:37:38.883546 30322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:37:38.883591 30322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.883661 30322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:37:38.883702 30322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:38.892192 30322 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.195:44227
I20260927 00:37:38.892215 30449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.195:44227 every 8 connection(s)
I20260927 00:37:38.893042 30322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/data/info.pb
I20260927 00:37:38.895696 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30322
I20260927 00:37:38.895813 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal/instance
I20260927 00:37:38.896132 10287 external_mini_cluster.cc:1428] Setting key d0c6290c39161e8924ce5bd4f0270d05
I20260927 00:37:38.902055 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.894790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:34141 (local address 127.10.11.254:43027)
0927 00:37:38.894935 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.894938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.895996 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 00:37:38.897408 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.897412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.897481 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.897581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:38.898518 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.899043 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.900159 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.900381 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.900490 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.901573 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:38.901577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:38.901579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:38.901599 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:38.901617 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:38.901624 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:38.901701 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.901862 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.901919 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:37:38.902837 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.895163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:43027 (local address 127.10.11.195:34141)
0927 00:37:38.895829 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.895883 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.897191 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.897757 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.897764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.897845 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:37:38.898381 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.898395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.899165 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.899169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.900033 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.900042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.900524 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.900528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.900696 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.901315 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:37:38.901317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:37:38.901354 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:37:38.901411 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:37:38.901715 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:37:38.901726 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:37:38.901733 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.901860 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.902153 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260927 00:37:38.903939 30450 heartbeater.cc:347] Connected to a master server at 127.10.11.254:43027
I20260927 00:37:38.904201 30450 heartbeater.cc:464] Registering TS with master...
I20260927 00:37:38.904627 30450 heartbeater.cc:511] Master 127.10.11.254:43027 requested a full tablet report, sending...
I20260927 00:37:38.905615 30000 ts_manager.cc:194] Registered new tserver with Master: 04a0bd8e9ff54a9b929614d1a12b8235 (127.10.11.195:44227)
I20260927 00:37:38.906214 30000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.11.195:34141
I20260927 00:37:38.909658 10287 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:37:38.914969 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.911493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.10.11.254:43027)
0927 00:37:38.911619 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.911622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.911778 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:37:38.911866 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.911868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.911901 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.911962 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:38.912208 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.912689 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.913579 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.913776 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.913852 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.914592 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:38.914595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:38.914597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:38.914607 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:38.914624 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:38.914631 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:38.914691 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.914818 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.914863 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:37:38.917582 30000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:37:38.921627 30000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:37:38.935416 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.930143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.195:44227 (local address 127.0.0.1:52670)
0927 00:37:38.930534 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.930545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.930649 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.931350 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.931353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.931378 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:38.931609 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.931615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.932579 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.932584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.933863 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.933872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.934848 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.934851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.934992 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.935028 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.935143 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.935255 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":171,"spinlock_wait_cycles":2432,"thread_start_us":86,"threads_started":1}
I20260927 00:37:38.935418 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.930542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.10.11.195:44227)
0927 00:37:38.930885 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.930900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.930921 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:37:38.930993 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.930998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.931194 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.931517 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:38.931704 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.932454 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.933970 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.934727 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.934846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.935040 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.935170 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.935258 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
I20260927 00:37:38.935822 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.929417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:35653 (local address 127.0.0.1:48586)
0927 00:37:38.929739 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.929783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.929897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.931486 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.931488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.931502 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:38.931672 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.931679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.932643 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.932646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.934221 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.934231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.935447 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.935451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.935536 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.935559 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.935627 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.935689 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":344,"thread_start_us":75,"threads_started":1}
I20260927 00:37:38.936265 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.929575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.10.11.194:35653)
0927 00:37:38.930885 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.930898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.930921 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:37:38.930993 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.930998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.931194 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.931345 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:38.931772 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.932490 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.934343 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.935135 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.935225 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.935386 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.936002 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.936100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":156,"threads_started":1}
I20260927 00:37:38.936477 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.929652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46083 (local address 127.0.0.1:54456)
0927 00:37:38.930360 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:38.930373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:38.930462 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:38.931337 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:38.931343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:38.931364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:38.931603 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.931614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.933872 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.933876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.934991 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:38.935002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.936075 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.936079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:38.936173 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.936197 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:37:38.936247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 00:37:38.936323 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":9,"spinlock_wait_cycles":3456,"thread_start_us":91,"threads_started":1}
I20260927 00:37:38.937283 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:38.930141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.10.11.193:46083)
0927 00:37:38.930957 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:38.930969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:38.930988 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:37:38.931047 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:38.931051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:38.931183 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:38.931345 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:38.932613 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.933754 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.935135 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:38.935926 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:38.936825 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:38.937000 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:38.937050 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:37:38.937136 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":139,"threads_started":1}
I20260927 00:37:38.938482 30384 tablet_service.cc:1467] Processing CreateTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c (DEFAULT_TABLE table=test-table [id=58bfe62677054e2cbbac3b703d96b01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:38.938599 30253 tablet_service.cc:1467] Processing CreateTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c (DEFAULT_TABLE table=test-table [id=58bfe62677054e2cbbac3b703d96b01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:38.939378 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ac3b461ac4ee98dfc83b90c00040c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:38.939378 30384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ac3b461ac4ee98dfc83b90c00040c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:38.942961 30122 tablet_service.cc:1467] Processing CreateTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c (DEFAULT_TABLE table=test-table [id=58bfe62677054e2cbbac3b703d96b01a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:37:38.943815 30122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ac3b461ac4ee98dfc83b90c00040c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:38.945288 30467 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Bootstrap starting.
I20260927 00:37:38.947479 30467 tablet_bootstrap.cc:654] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:38.948213 30468 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Bootstrap starting.
I20260927 00:37:38.948334 30467 log.cc:824] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:38.950306 30468 tablet_bootstrap.cc:654] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:38.950385 30467 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: No bootstrap required, opened a new log
I20260927 00:37:38.950387 30470 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Bootstrap starting.
I20260927 00:37:38.950659 30467 ts_tablet_manager.cc:1397] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:37:38.951202 30468 log.cc:824] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:38.952379 30470 tablet_bootstrap.cc:654] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:38.953006 30470 log.cc:824] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:38.953325 30468 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: No bootstrap required, opened a new log
I20260927 00:37:38.953550 30468 ts_tablet_manager.cc:1397] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:37:38.954802 30470 tablet_bootstrap.cc:492] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: No bootstrap required, opened a new log
I20260927 00:37:38.954984 30470 ts_tablet_manager.cc:1397] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:37:38.954941 30467 raft_consensus.cc:348] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.955224 30467 raft_consensus.cc:374] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:38.955288 30467 raft_consensus.cc:729] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a0bd8e9ff54a9b929614d1a12b8235, State: Initialized, Role: FOLLOWER
I20260927 00:37:38.955790 30467 consensus_queue.cc:260] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.956475 30450 heartbeater.cc:503] Master 127.10.11.254:43027 was elected leader, sending a full tablet report...
I20260927 00:37:38.956727 30467 ts_tablet_manager.cc:1428] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:37:38.957469 30468 raft_consensus.cc:348] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.957682 30468 raft_consensus.cc:374] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:38.958539 30468 raft_consensus.cc:729] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499b93e970bf4883aebde93ff8376954, State: Initialized, Role: FOLLOWER
I20260927 00:37:38.958861 30470 raft_consensus.cc:348] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.959163 30470 raft_consensus.cc:374] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:38.959259 30470 raft_consensus.cc:729] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb44e086f7d409b9cea437db210d0d4, State: Initialized, Role: FOLLOWER
I20260927 00:37:38.959369 30468 consensus_queue.cc:260] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.959668 30470 consensus_queue.cc:260] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:38.960011 30319 heartbeater.cc:503] Master 127.10.11.254:43027 was elected leader, sending a full tablet report...
I20260927 00:37:38.960332 30468 ts_tablet_manager.cc:1428] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:37:38.961302 30470 ts_tablet_manager.cc:1428] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:37:38.961647 30188 heartbeater.cc:503] Master 127.10.11.254:43027 was elected leader, sending a full tablet report...
W20260927 00:37:39.007560 30189 tablet.cc:2369] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:37:39.030027 30473 raft_consensus.cc:482] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:37:39.030349 30473 raft_consensus.cc:504] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:39.031523 30473 leader_election.cc:288] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499b93e970bf4883aebde93ff8376954 (127.10.11.194:35653), ecb44e086f7d409b9cea437db210d0d4 (127.10.11.193:46083)
I20260927 00:37:39.035449 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.031745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.194:35653 (local address 127.10.11.195:56071)
0927 00:37:39.031934 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:39.031953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:39.032045 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:39.032388 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:39.032391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:39.032444 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:39.032612 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:39.032624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.033436 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.033440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:39.034373 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:39.034382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.035027 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.035031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:39.035141 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:39.035165 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:37:39.035235 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:37:39.035304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":90}
I20260927 00:37:39.035477 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.031852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:56071 (local address 127.10.11.194:35653)
0927 00:37:39.031984 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:39.031991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:39.032007 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:37:39.032195 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:39.032201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:39.032261 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:39.032338 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:39.032748 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.033271 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.034541 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.034910 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.035012 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:39.035049 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:39.035262 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 00:37:39.035341 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:37:39.035615 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.031766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46083 (local address 127.10.11.195:44299)
0927 00:37:39.032145 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:39.032160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:37:39.032259 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:37:39.032557 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:37:39.032559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:37:39.032574 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:37:39.032712 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:39.032718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.033460 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.033466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:39.034366 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:37:39.034375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.035182 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.035185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:37:39.035295 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:39.035317 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:37:39.035394 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:37:39.035466 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":16,"thread_start_us":120,"threads_started":1}
I20260927 00:37:39.035629 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.031889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.11.195:44299 (local address 127.10.11.193:46083)
0927 00:37:39.032138 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:39.032145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:39.032243 (+ 98us) server_negotiation.cc:407] Connection header received
0927 00:37:39.032399 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:39.032402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:39.032454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:39.032541 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:37:39.032827 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.033306 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.034485 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.035058 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.035170 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:39.035230 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:39.035427 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 00:37:39.035503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 00:37:39.036060 30273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ac3b461ac4ee98dfc83b90c00040c" candidate_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499b93e970bf4883aebde93ff8376954" is_pre_election: true
I20260927 00:37:39.036154 30142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ac3b461ac4ee98dfc83b90c00040c" candidate_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb44e086f7d409b9cea437db210d0d4" is_pre_election: true
I20260927 00:37:39.036384 30273 raft_consensus.cc:2454] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a0bd8e9ff54a9b929614d1a12b8235 in term 0.
I20260927 00:37:39.036427 30142 raft_consensus.cc:2454] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a0bd8e9ff54a9b929614d1a12b8235 in term 0.
I20260927 00:37:39.036857 30340 leader_election.cc:302] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a0bd8e9ff54a9b929614d1a12b8235, ecb44e086f7d409b9cea437db210d0d4; no voters:
I20260927 00:37:39.037148 30473 raft_consensus.cc:2781] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:37:39.037230 30473 raft_consensus.cc:482] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:37:39.037271 30473 raft_consensus.cc:3037] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:39.038268 30473 raft_consensus.cc:504] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:39.038612 30473 leader_election.cc:288] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [CANDIDATE]: Term 1 election: Requested vote from peers 499b93e970bf4883aebde93ff8376954 (127.10.11.194:35653), ecb44e086f7d409b9cea437db210d0d4 (127.10.11.193:46083)
I20260927 00:37:39.038877 30273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ac3b461ac4ee98dfc83b90c00040c" candidate_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499b93e970bf4883aebde93ff8376954"
I20260927 00:37:39.038949 30142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ac3b461ac4ee98dfc83b90c00040c" candidate_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb44e086f7d409b9cea437db210d0d4"
I20260927 00:37:39.039011 30273 raft_consensus.cc:3037] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:39.039083 30142 raft_consensus.cc:3037] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:39.040160 30273 raft_consensus.cc:2454] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a0bd8e9ff54a9b929614d1a12b8235 in term 1.
I20260927 00:37:39.040167 30142 raft_consensus.cc:2454] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a0bd8e9ff54a9b929614d1a12b8235 in term 1.
I20260927 00:37:39.040481 30339 leader_election.cc:302] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a0bd8e9ff54a9b929614d1a12b8235, 499b93e970bf4883aebde93ff8376954; no voters:
I20260927 00:37:39.040654 30473 raft_consensus.cc:2781] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:39.040838 30473 raft_consensus.cc:686] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 1 LEADER]: Becoming Leader. State: Replica: 04a0bd8e9ff54a9b929614d1a12b8235, State: Running, Role: LEADER
I20260927 00:37:39.041204 30473 consensus_queue.cc:237] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } }
I20260927 00:37:39.044200 29999 catalog_manager.cc:5495] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04a0bd8e9ff54a9b929614d1a12b8235 (127.10.11.195). New cstate: current_term: 1 leader_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a0bd8e9ff54a9b929614d1a12b8235" member_type: VOTER last_known_addr { host: "127.10.11.195" port: 44227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20260927 00:37:39.055879 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.052804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.10.11.195:44227)
0927 00:37:39.052911 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:39.052915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:39.052926 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:37:39.053045 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:39.053047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:39.053091 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:39.053173 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:37:39.053357 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.053812 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.054756 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:39.054951 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:39.055026 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:39.055240 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:37:39.055586 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:37:39.055677 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:39.055739 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:37:39.055776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:37:39.060986 30273 raft_consensus.cc:1261] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 1 FOLLOWER]: Refusing update from remote peer 04a0bd8e9ff54a9b929614d1a12b8235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:39.060986 30142 raft_consensus.cc:1261] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4 [term 1 FOLLOWER]: Refusing update from remote peer 04a0bd8e9ff54a9b929614d1a12b8235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:37:39.061655 30477 consensus_queue.cc:1035] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499b93e970bf4883aebde93ff8376954" member_type: VOTER last_known_addr { host: "127.10.11.194" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:39.061826 30473 consensus_queue.cc:1035] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb44e086f7d409b9cea437db210d0d4" member_type: VOTER last_known_addr { host: "127.10.11.193" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:37:39.069160 30482 mvcc.cc:205] Tried to move back new op lower bound from 7333762904307896320 to 7333762904235716608. Current Snapshot: MvccSnapshot[applied={T|T < 7333762904307896320}]
I20260927 00:37:39.076552 30000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:39.076713 30000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:37:39.079123 30000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb81a609461d42fda2016a2028379f07: Sending DeleteTablet for 3 replicas of tablet e22ac3b461ac4ee98dfc83b90c00040c
I20260927 00:37:39.079981 30122 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:39 UTC) from {username='slave'} at 127.0.0.1:54456
I20260927 00:37:39.079963 30384 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:39 UTC) from {username='slave'} at 127.0.0.1:52670
I20260927 00:37:39.080437 30253 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ac3b461ac4ee98dfc83b90c00040c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:37:39 UTC) from {username='slave'} at 127.0.0.1:48586
I20260927 00:37:39.080955 30489 tablet_replica.cc:330] T e22ac3b461ac4ee98dfc83b90c00040c P ecb44e086f7d409b9cea437db210d0d4: stopping tablet replica
I20260927 00:37:39.081027 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30060
I20260927 00:37:39.081084 30491 tablet_replica.cc:330] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: stopping tablet replica
I20260927 00:37:39.082410 30490 tablet_replica.cc:330] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: stopping tablet replica
I20260927 00:37:39.082690 30490 raft_consensus.cc:2229] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:37:39.083053 30490 raft_consensus.cc:2258] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:39.083508 30491 raft_consensus.cc:2229] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:37:39.083858 30491 raft_consensus.cc:2258] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:37:39.084649 30490 ts_tablet_manager.cc:1905] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:39.085263 30491 ts_tablet_manager.cc:1905] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:37:39.087594 30490 ts_tablet_manager.cc:1918] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:39.087738 30490 log.cc:1196] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-2/wal/wals/e22ac3b461ac4ee98dfc83b90c00040c
I20260927 00:37:39.088147 30490 ts_tablet_manager.cc:1939] T e22ac3b461ac4ee98dfc83b90c00040c P 04a0bd8e9ff54a9b929614d1a12b8235: Deleting consensus metadata
I20260927 00:37:39.088318 30491 ts_tablet_manager.cc:1918] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:37:39.088486 30491 log.cc:1196] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Deleting WAL directory at /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469213433099-10287-0/minicluster-data/ts-1/wal/wals/e22ac3b461ac4ee98dfc83b90c00040c
I20260927 00:37:39.088877 30491 ts_tablet_manager.cc:1939] T e22ac3b461ac4ee98dfc83b90c00040c P 499b93e970bf4883aebde93ff8376954: Deleting consensus metadata
W20260927 00:37:39.089326 29987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.11.193:46083 (error 108)
I20260927 00:37:39.089449 29986 catalog_manager.cc:4841] TS 04a0bd8e9ff54a9b929614d1a12b8235 (127.10.11.195:44227): tablet e22ac3b461ac4ee98dfc83b90c00040c (table test-table [id=58bfe62677054e2cbbac3b703d96b01a]) successfully deleted
I20260927 00:37:39.090039 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30191
I20260927 00:37:39.090421 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:39.090060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.193:46083 (local address 127.0.0.1:54470)
0927 00:37:39.090257 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 00:37:39.090334 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 00:37:39.090579 29987 catalog_manager.cc:4584] TS ecb44e086f7d409b9cea437db210d0d4 (127.10.11.193:46083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e22ac3b461ac4ee98dfc83b90c00040c: Network error: Client connection negotiation failed: client connection to 127.10.11.193:46083: connect: Connection refused (error 111)
I20260927 00:37:39.097069 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30322
I20260927 00:37:39.101806 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 29969
2026-09-27T00:37:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1017 ms)
[----------] 6 tests from EncryptionPolicyTest (10429 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:39 dist-test-slave-6gxc krb5kdc[30495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:39 dist-test-slave-6gxc krb5kdc[30495](info): set up 2 sockets
Sep 27 00:37:39 dist-test-slave-6gxc krb5kdc[30495](info): commencing operation
krb5kdc: starting...
W20260927 00:37:40.138425 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:40.257608 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:40.365890 30508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:40.366211 30508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:40.366328 30508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:40.370245 30508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:40.370332 30508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:40.370386 30508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:40.370440 30508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:40.370479 30508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:40.375537 30508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:40.376502 30508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:40.377458 30508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:40.385509 30516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:40.385542 30513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:40.385520 30514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:40.385790 30508 server_base.cc:1034] running on GCE node
I20260927 00:37:40.386533 30508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:40.387512 30508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:40.388679 30508 hybrid_clock.cc:648] HybridClock initialized: now 1790469460388651 us; error 41 us; skew 500 ppm
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:40.391304 30508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:40.392099 30508 webserver.cc:460] Webserver started at http://127.0.0.1:42385/ using document root <none> and password file <none>
I20260927 00:37:40.392685 30508 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:40.392771 30508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:40.393002 30508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:40.394924 30508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "2068f7c351564e1ca64746306f0aaca7"
format_stamp: "Formatted at 2026-09-27 00:37:40 on dist-test-slave-6gxc"
server_key: "515f5ce03962f58de0a2a4462a623403"
server_key_iv: "eca5b530c72eaa7a33d539ba8ce221a0"
server_key_version: "encryptionkey@0"
I20260927 00:37:40.395459 30508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "2068f7c351564e1ca64746306f0aaca7"
format_stamp: "Formatted at 2026-09-27 00:37:40 on dist-test-slave-6gxc"
server_key: "515f5ce03962f58de0a2a4462a623403"
server_key_iv: "eca5b530c72eaa7a33d539ba8ce221a0"
server_key_version: "encryptionkey@0"
I20260927 00:37:40.398900 30508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:40.401065 30523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:40.402450 30508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:40.402583 30508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "2068f7c351564e1ca64746306f0aaca7"
format_stamp: "Formatted at 2026-09-27 00:37:40 on dist-test-slave-6gxc"
server_key: "515f5ce03962f58de0a2a4462a623403"
server_key_iv: "eca5b530c72eaa7a33d539ba8ce221a0"
server_key_version: "encryptionkey@0"
I20260927 00:37:40.402696 30508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:40.419086 30508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:40.421689 30508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:40.421880 30508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:40.428527 30508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42937
I20260927 00:37:40.428552 30575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42937 every 8 connection(s)
I20260927 00:37:40.429368 30508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:40.431859 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:40.433247 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30508
I20260927 00:37:40.433403 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:40.433700 10287 external_mini_cluster.cc:1428] Setting key 7b7576ca1348dfa7ca888e6c00481e29
I20260927 00:37:40.437553 30576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: Bootstrap starting.
I20260927 00:37:40.439613 30576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:40.440274 30576 log.cc:824] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:37:40.442317 30576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: No bootstrap required, opened a new log
I20260927 00:37:40.446771 30576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260927 00:37:40.447033 30576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:40.447101 30576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2068f7c351564e1ca64746306f0aaca7, State: Initialized, Role: FOLLOWER
I20260927 00:37:40.447530 30576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260927 00:37:40.447672 30576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:40.447734 30576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:40.447748 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:40.435025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33377 (local address 127.0.0.1:42937)
0927 00:37:40.435428 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:40.435442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:40.435474 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:37:40.436066 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:40.436081 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:40.436376 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:40.436697 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:40.437478 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:40.438517 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:40.439372 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:40.439881 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:40.440043 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:40.442836 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:40.442842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:40.442857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:40.442884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:40.444814 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:40.445284 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:40.445290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:40.445295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:40.445431 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:40.445742 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:40.445745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:40.445747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:40.446233 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:40.446447 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:40.446633 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:37:40.446831 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260927 00:37:40.447845 30576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:40.448812 30576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260927 00:37:40.449153 30576 leader_election.cc:302] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2068f7c351564e1ca64746306f0aaca7; no voters:
I20260927 00:37:40.449448 30576 leader_election.cc:288] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:40.449512 30581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:40.449692 30581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [term 1 LEADER]: Becoming Leader. State: Replica: 2068f7c351564e1ca64746306f0aaca7, State: Running, Role: LEADER
I20260927 00:37:40.450001 30581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260927 00:37:40.450604 30576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:40.450767 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:40.451542 30583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2068f7c351564e1ca64746306f0aaca7. Latest consensus state: current_term: 1 leader_uuid: "2068f7c351564e1ca64746306f0aaca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } } }
I20260927 00:37:40.451663 30583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:40.451970 30582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2068f7c351564e1ca64746306f0aaca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2068f7c351564e1ca64746306f0aaca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } } }
I20260927 00:37:40.452044 30582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7 [sys.catalog]: This master's current role is: LEADER
W20260927 00:37:40.455482 30601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:37:40.455560 30601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:37:40.455618 30590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:40.456439 30590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:40.461043 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:40.453274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43307 (local address 127.0.0.1:42937)
0927 00:37:40.453385 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:40.453389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:40.453629 (+ 240us) server_negotiation.cc:407] Connection header received
0927 00:37:40.453767 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:40.453770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:40.453808 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:40.453905 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:40.454562 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:40.455052 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:40.455959 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:40.456208 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:40.456310 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:40.457604 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:40.457608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:40.457610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:40.457632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:40.459206 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:40.459739 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:40.459743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:40.459745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:40.459819 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:40.460232 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:40.460236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:40.460237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:40.460423 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:40.460560 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:40.460818 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:37:40.460913 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:37:40.461997 30590 catalog_manager.cc:1347] Generated new cluster ID: f6aef11c9ff740889fff74b5170961b6
I20260927 00:37:40.462132 30590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:40.475908 30590 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:40.476742 30590 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:40.481274 30590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2068f7c351564e1ca64746306f0aaca7: Generated new TSK 0
I20260927 00:37:40.481940 30590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:40.485620 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30508
W20260927 00:37:40.490907 30493 connection.cc:535] client connection to 127.0.0.1:42937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42937): Connection reset by peer (error 104)
2026-09-27T00:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30607](info): set up 2 sockets
Sep 27 00:37:40 dist-test-slave-6gxc krb5kdc[30607](info): commencing operation
krb5kdc: starting...
W20260927 00:37:41.534219 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:41 dist-test-slave-6gxc krb5kdc[30607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:41.646934 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:41.751737 30620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:41.752019 30620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:41.752099 30620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:41.756084 30620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:41.756174 30620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:41.756211 30620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:41.756266 30620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:41.756304 30620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:41.761346 30620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:41.762346 30620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:41.763270 30620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:41.771582 30626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:41.771564 30628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:41.771636 30620 server_base.cc:1034] running on GCE node
W20260927 00:37:41.771562 30625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:41.772408 30620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:41.773391 30620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:41.774566 30620 hybrid_clock.cc:648] HybridClock initialized: now 1790469461774541 us; error 36 us; skew 500 ppm
Sep 27 00:37:41 dist-test-slave-6gxc krb5kdc[30607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:41.777292 30620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:41.778148 30620 webserver.cc:460] Webserver started at http://127.0.0.1:39879/ using document root <none> and password file <none>
I20260927 00:37:41.778685 30620 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:41.778745 30620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:41.778990 30620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:41.780949 30620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "3d9196ad38bf49efb74e5433d04133d3"
format_stamp: "Formatted at 2026-09-27 00:37:41 on dist-test-slave-6gxc"
server_key: "0bf065bb8a43d2f86cc95acf801175ed"
server_key_iv: "31d1427faac1d53eebce9dd3a9f62333"
server_key_version: "encryptionkey@0"
I20260927 00:37:41.781500 30620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "3d9196ad38bf49efb74e5433d04133d3"
format_stamp: "Formatted at 2026-09-27 00:37:41 on dist-test-slave-6gxc"
server_key: "0bf065bb8a43d2f86cc95acf801175ed"
server_key_iv: "31d1427faac1d53eebce9dd3a9f62333"
server_key_version: "encryptionkey@0"
I20260927 00:37:41.785171 30620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:37:41.787545 30635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:41.788750 30620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:37:41.788875 30620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "3d9196ad38bf49efb74e5433d04133d3"
format_stamp: "Formatted at 2026-09-27 00:37:41 on dist-test-slave-6gxc"
server_key: "0bf065bb8a43d2f86cc95acf801175ed"
server_key_iv: "31d1427faac1d53eebce9dd3a9f62333"
server_key_version: "encryptionkey@0"
I20260927 00:37:41.788986 30620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:41.805548 30620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:41.808634 30620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:41.808830 30620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:41.815492 30620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44401
I20260927 00:37:41.815507 30687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44401 every 8 connection(s)
I20260927 00:37:41.816437 30620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:41.819132 30688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:41.822305 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30620
I20260927 00:37:41.822456 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:41.822750 10287 external_mini_cluster.cc:1428] Setting key 21da4f91a069f8d246e370e5aa3b5fc7
I20260927 00:37:41.824775 30688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: Bootstrap starting.
I20260927 00:37:41.826990 30688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:41.827844 30688 log.cc:824] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:41.829982 30688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: No bootstrap required, opened a new log
I20260927 00:37:41.831955 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:41.824068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46707 (local address 127.0.0.1:44401)
0927 00:37:41.824443 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:41.824458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:41.824492 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:37:41.825135 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:41.825152 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:41.825468 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:41.825732 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:41.826173 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:41.827202 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:41.828410 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:41.828966 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:41.829089 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:41.830085 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:41.830093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:41.830111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:41.830516 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:41.830566 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:41.830589 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:41.830692 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:41.830942 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:37:41.831080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260927 00:37:41.832971 30688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260927 00:37:41.833216 30688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:41.833274 30688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9196ad38bf49efb74e5433d04133d3, State: Initialized, Role: FOLLOWER
I20260927 00:37:41.833700 30688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260927 00:37:41.833876 30688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:41.833948 30688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:41.834038 30688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:41.834872 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:41.835098 30688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260927 00:37:41.835439 30688 leader_election.cc:302] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d9196ad38bf49efb74e5433d04133d3; no voters:
I20260927 00:37:41.835786 30688 leader_election.cc:288] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:41.835877 30693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:41.836113 30693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3d9196ad38bf49efb74e5433d04133d3, State: Running, Role: LEADER
I20260927 00:37:41.836440 30693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260927 00:37:41.837020 30688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:41.837595 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d9196ad38bf49efb74e5433d04133d3. Latest consensus state: current_term: 1 leader_uuid: "3d9196ad38bf49efb74e5433d04133d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } } }
I20260927 00:37:41.837760 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:41.838011 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d9196ad38bf49efb74e5433d04133d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d9196ad38bf49efb74e5433d04133d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } } }
I20260927 00:37:41.838119 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:41.841856 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:41.837061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41191 (local address 127.0.0.1:44401)
0927 00:37:41.837152 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:41.837156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:41.837546 (+ 390us) server_negotiation.cc:407] Connection header received
0927 00:37:41.837712 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:41.837716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:41.837765 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:41.837823 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:41.838309 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:41.838890 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:41.839939 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:41.840282 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:41.840378 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:41.841263 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:41.841267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:41.841270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:41.841281 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:41.841306 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:41.841318 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:41.841392 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:41.841707 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 00:37:41.841737 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:37:41.842185 30713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:37:41.842249 30713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:37:41.842326 30704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:41.843184 30704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:41.848327 30704 catalog_manager.cc:1347] Generated new cluster ID: fc6663afa88b4802907a57026c7729a7
I20260927 00:37:41.848400 30704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:41.863257 30704 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:41.864830 30704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:41.872756 30704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d9196ad38bf49efb74e5433d04133d3: Generated new TSK 0
I20260927 00:37:41.873615 30704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:41.884632 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30620
W20260927 00:37:41.889789 30605 connection.cc:535] client connection to 127.0.0.1:44401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44401): Connection reset by peer (error 104)
2026-09-27T00:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:41 dist-test-slave-6gxc krb5kdc[30719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:41 dist-test-slave-6gxc krb5kdc[30719](info): set up 2 sockets
Sep 27 00:37:41 dist-test-slave-6gxc krb5kdc[30719](info): commencing operation
krb5kdc: starting...
W20260927 00:37:42.918426 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:43 dist-test-slave-6gxc krb5kdc[30719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:43.057705 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:43.165076 30732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:43.165306 30732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:43.165345 30732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:43.169183 30732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:43.169232 30732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:43.169251 30732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:43.169267 30732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:43.169282 30732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:43.177206 30732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:43.178866 30732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:43.179940 30732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:43.188243 30740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:43.188246 30738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:43.188308 30737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:43.188586 30732 server_base.cc:1034] running on GCE node
I20260927 00:37:43.189168 30732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:43.190243 30732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:43.191401 30732 hybrid_clock.cc:648] HybridClock initialized: now 1790469463191389 us; error 23 us; skew 500 ppm
Sep 27 00:37:43 dist-test-slave-6gxc krb5kdc[30719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:43.194018 30732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:43.194840 30732 webserver.cc:460] Webserver started at http://127.0.0.1:43103/ using document root <none> and password file <none>
I20260927 00:37:43.195367 30732 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:43.195426 30732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:43.195647 30732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:43.197552 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "3b6c837a8f8c44e580b9dbfda1434b3a"
format_stamp: "Formatted at 2026-09-27 00:37:43 on dist-test-slave-6gxc"
server_key: "2f98c48c5670a9b0aec3fb162574e555"
server_key_iv: "3b134647c62f3d896a82dbf954e3db0f"
server_key_version: "encryptionkey@0"
I20260927 00:37:43.198081 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "3b6c837a8f8c44e580b9dbfda1434b3a"
format_stamp: "Formatted at 2026-09-27 00:37:43 on dist-test-slave-6gxc"
server_key: "2f98c48c5670a9b0aec3fb162574e555"
server_key_iv: "3b134647c62f3d896a82dbf954e3db0f"
server_key_version: "encryptionkey@0"
I20260927 00:37:43.201531 30732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:37:43.203840 30747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:43.204962 30732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:37:43.205112 30732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "3b6c837a8f8c44e580b9dbfda1434b3a"
format_stamp: "Formatted at 2026-09-27 00:37:43 on dist-test-slave-6gxc"
server_key: "2f98c48c5670a9b0aec3fb162574e555"
server_key_iv: "3b134647c62f3d896a82dbf954e3db0f"
server_key_version: "encryptionkey@0"
I20260927 00:37:43.205219 30732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:43.209507 30732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:43.212383 30732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:43.212560 30732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:43.219414 30732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45693
I20260927 00:37:43.219447 30799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45693 every 8 connection(s)
I20260927 00:37:43.220261 30732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:43.223031 30800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:43.223119 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30732
I20260927 00:37:43.223263 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:43.223568 10287 external_mini_cluster.cc:1428] Setting key 05b2eea67c5a839a84e9d13c0f5ecf7f
I20260927 00:37:43.228621 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:43.224800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48075 (local address 127.0.0.1:45693)
0927 00:37:43.225204 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:43.225217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:43.225250 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:43.225962 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:43.225979 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:43.226387 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:43.226680 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:43.227499 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:43.227503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:43.227513 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:43.227523 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:43.227568 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:43.227586 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:43.227692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:43.227799 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:37:43.227953 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260927 00:37:43.229527 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a: Bootstrap starting.
I20260927 00:37:43.231173 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:43.231861 30800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:43.232738 30800 log.cc:824] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:43.234975 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a: No bootstrap required, opened a new log
I20260927 00:37:43.234997 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:43.233395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49453 (local address 127.0.0.1:45693)
0927 00:37:43.233504 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:43.233508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:43.233744 (+ 236us) server_negotiation.cc:407] Connection header received
0927 00:37:43.233845 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:43.233848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:43.233874 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:43.233936 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:43.234581 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:43.234584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:43.234586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:43.234597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:43.234619 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:43.234629 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:43.234705 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:43.234823 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 00:37:43.234852 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:37:43.238525 30800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260927 00:37:43.238718 30800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:43.238777 30800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c837a8f8c44e580b9dbfda1434b3a, State: Initialized, Role: FOLLOWER
I20260927 00:37:43.239225 30800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260927 00:37:43.239364 30800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:43.239428 30800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:43.239526 30800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:43.240347 30800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260927 00:37:43.240674 30800 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b6c837a8f8c44e580b9dbfda1434b3a; no voters:
I20260927 00:37:43.240950 30800 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:43.241101 30810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:43.241341 30810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [term 1 LEADER]: Becoming Leader. State: Replica: 3b6c837a8f8c44e580b9dbfda1434b3a, State: Running, Role: LEADER
I20260927 00:37:43.241644 30810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260927 00:37:43.242059 30800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:43.243315 30811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260927 00:37:43.243311 30812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6c837a8f8c44e580b9dbfda1434b3a. Latest consensus state: current_term: 1 leader_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c837a8f8c44e580b9dbfda1434b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260927 00:37:43.243464 30812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:43.243464 30811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:43.243897 30821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:43.246742 30821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:43.251284 30821 catalog_manager.cc:1347] Generated new cluster ID: f4199b1f6c2b47998581458bf395037d
I20260927 00:37:43.251341 30821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:43.264123 30821 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:43.264395 30821 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:43.271006 30821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6c837a8f8c44e580b9dbfda1434b3a: Generated new TSK 0
I20260927 00:37:43.271566 30821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:43.278496 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30732
2026-09-27T00:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:43 dist-test-slave-6gxc krb5kdc[30831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:43 dist-test-slave-6gxc krb5kdc[30831](info): set up 2 sockets
Sep 27 00:37:43 dist-test-slave-6gxc krb5kdc[30831](info): commencing operation
krb5kdc: starting...
W20260927 00:37:44.310309 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:44.426472 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:44.529206 30844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:44.529470 30844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:44.529587 30844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:44.533471 30844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:44.533550 30844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:44.533588 30844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:44.533643 30844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:44.533680 30844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:44.538640 30844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:44.539520 30844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:44.540400 30844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:44.548787 30849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:44.548811 30852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:44.548997 30844 server_base.cc:1034] running on GCE node
W20260927 00:37:44.548789 30850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:44.549691 30844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:44.550669 30844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:44.551822 30844 hybrid_clock.cc:648] HybridClock initialized: now 1790469464551801 us; error 32 us; skew 500 ppm
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:44.558393 30844 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:44.559201 30844 webserver.cc:460] Webserver started at http://127.0.0.1:46067/ using document root <none> and password file <none>
I20260927 00:37:44.559718 30844 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:44.559777 30844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:44.559968 30844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:44.561839 30844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "f13da56e03fb400db9a9811f3a247246"
format_stamp: "Formatted at 2026-09-27 00:37:44 on dist-test-slave-6gxc"
server_key: "34629296c370768862bc9a1f48bb6681"
server_key_iv: "883d04955bdb7a38fc779eaf6d44a511"
server_key_version: "encryptionkey@0"
I20260927 00:37:44.562429 30844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "f13da56e03fb400db9a9811f3a247246"
format_stamp: "Formatted at 2026-09-27 00:37:44 on dist-test-slave-6gxc"
server_key: "34629296c370768862bc9a1f48bb6681"
server_key_iv: "883d04955bdb7a38fc779eaf6d44a511"
server_key_version: "encryptionkey@0"
I20260927 00:37:44.565829 30844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:37:44.568277 30859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:44.569391 30844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:44.569485 30844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "f13da56e03fb400db9a9811f3a247246"
format_stamp: "Formatted at 2026-09-27 00:37:44 on dist-test-slave-6gxc"
server_key: "34629296c370768862bc9a1f48bb6681"
server_key_iv: "883d04955bdb7a38fc779eaf6d44a511"
server_key_version: "encryptionkey@0"
I20260927 00:37:44.569607 30844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:44.593389 30844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:44.596567 30844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:44.596767 30844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:44.603488 30844 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41889
I20260927 00:37:44.603515 30911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41889 every 8 connection(s)
I20260927 00:37:44.604355 30844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:44.606945 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:44.611742 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30844
I20260927 00:37:44.611876 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:44.612133 10287 external_mini_cluster.cc:1428] Setting key 1e48b8bce95a5ca24896b03562914cab
I20260927 00:37:44.612205 30912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246: Bootstrap starting.
I20260927 00:37:44.614590 30912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:44.615283 30912 log.cc:824] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:44.617401 30912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246: No bootstrap required, opened a new log
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:37:44.621186 30912 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260927 00:37:44.621433 30912 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:44.621551 30912 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13da56e03fb400db9a9811f3a247246, State: Initialized, Role: FOLLOWER
I20260927 00:37:44.622040 30912 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260927 00:37:44.622243 30912 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:44.622321 30912 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:44.622431 30912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:44.623602 30912 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260927 00:37:44.623970 30912 leader_election.cc:302] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13da56e03fb400db9a9811f3a247246; no voters:
I20260927 00:37:44.624332 30912 leader_election.cc:288] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:44.624420 30917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:44.624681 30917 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [term 1 LEADER]: Becoming Leader. State: Replica: f13da56e03fb400db9a9811f3a247246, State: Running, Role: LEADER
I20260927 00:37:44.625011 30917 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260927 00:37:44.625597 30912 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:44.626227 30917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13da56e03fb400db9a9811f3a247246. Latest consensus state: current_term: 1 leader_uuid: "f13da56e03fb400db9a9811f3a247246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } } }
I20260927 00:37:44.626374 30917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:44.626761 30921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:44.627177 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:44.613372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57649 (local address 127.0.0.1:41889)
0927 00:37:44.613758 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:44.613772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:44.613805 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:44.614446 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:44.614462 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:44.614830 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:44.615157 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:44.615992 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:44.617415 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:44.618365 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:44.619046 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:44.619205 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:44.621607 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:44.621615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:44.621633 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:44.621663 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:44.623700 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:44.624244 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:44.624250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:44.624256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:44.624420 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:44.624773 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:44.624777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:44.624778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:44.625248 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:44.625412 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:44.625918 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 00:37:44.626351 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260927 00:37:44.627808 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13da56e03fb400db9a9811f3a247246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13da56e03fb400db9a9811f3a247246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } } }
I20260927 00:37:44.627924 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:44.630081 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:44.630618 30921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:44.636227 30921 catalog_manager.cc:1347] Generated new cluster ID: 3a73bfd22aa14334988bffd4d6ebbb65
I20260927 00:37:44.636313 30921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:44.642302 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:44.632222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43273 (local address 127.0.0.1:41889)
0927 00:37:44.632326 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:44.632329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:44.632553 (+ 224us) server_negotiation.cc:407] Connection header received
0927 00:37:44.632726 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:44.632729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:44.632772 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:44.632821 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:44.633697 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:44.634496 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:44.635293 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:44.635569 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:44.635682 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:44.639045 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:44.639049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:44.639052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:44.639078 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:44.640673 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:44.641114 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:44.641117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:44.641119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:44.641203 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:44.641465 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:44.641469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:44.641470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:44.641666 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:44.641788 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:44.641998 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 00:37:44.642157 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:37:44.655535 30921 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:44.656436 30921 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:44.661378 30921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13da56e03fb400db9a9811f3a247246: Generated new TSK 0
I20260927 00:37:44.661991 30921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:44.666765 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30844
2026-09-27T00:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30943](info): set up 2 sockets
Sep 27 00:37:44 dist-test-slave-6gxc krb5kdc[30943](info): commencing operation
krb5kdc: starting...
W20260927 00:37:45.694090 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:45 dist-test-slave-6gxc krb5kdc[30943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:45.807279 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:45.912668 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:45.912897 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:45.912937 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:45.916846 30956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:45.916898 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:45.916917 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:45.916934 30956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:45.916987 30956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:45.922133 30956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:45.923065 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:45.924003 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:45.932289 30962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:45.932289 30961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:45.932322 30964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:45.932516 30956 server_base.cc:1034] running on GCE node
I20260927 00:37:45.933128 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:45.934237 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:45.935436 30956 hybrid_clock.cc:648] HybridClock initialized: now 1790469465935423 us; error 36 us; skew 500 ppm
Sep 27 00:37:45 dist-test-slave-6gxc krb5kdc[30943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:45.937985 30956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:45.938834 30956 webserver.cc:460] Webserver started at http://127.0.0.1:42785/ using document root <none> and password file <none>
I20260927 00:37:45.939366 30956 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:45.939455 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:45.939702 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:45.941609 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "e15f8ee4196e415289afe75a34559579"
format_stamp: "Formatted at 2026-09-27 00:37:45 on dist-test-slave-6gxc"
server_key: "a45808164d2d2cce18696aaa3160f26e"
server_key_iv: "c1bc2e976195d1000562e7da9ec4a516"
server_key_version: "encryptionkey@0"
I20260927 00:37:45.942220 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "e15f8ee4196e415289afe75a34559579"
format_stamp: "Formatted at 2026-09-27 00:37:45 on dist-test-slave-6gxc"
server_key: "a45808164d2d2cce18696aaa3160f26e"
server_key_iv: "c1bc2e976195d1000562e7da9ec4a516"
server_key_version: "encryptionkey@0"
I20260927 00:37:45.946127 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:37:45.948923 30971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:45.950139 30956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:45.950273 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "e15f8ee4196e415289afe75a34559579"
format_stamp: "Formatted at 2026-09-27 00:37:45 on dist-test-slave-6gxc"
server_key: "a45808164d2d2cce18696aaa3160f26e"
server_key_iv: "c1bc2e976195d1000562e7da9ec4a516"
server_key_version: "encryptionkey@0"
I20260927 00:37:45.950382 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:45.961969 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:45.965122 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:45.965345 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:45.972828 30956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38231
I20260927 00:37:45.972854 31023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38231 every 8 connection(s)
I20260927 00:37:45.973726 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:45.976387 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:45.981796 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: Bootstrap starting.
I20260927 00:37:45.982618 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 30956
I20260927 00:37:45.982754 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:45.983037 10287 external_mini_cluster.cc:1428] Setting key 8e72223c670706e4324340801b4ad844
I20260927 00:37:45.984498 31024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:45.985237 31024 log.cc:824] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:45.987218 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: No bootstrap required, opened a new log
I20260927 00:37:45.991506 31024 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 00:37:45.991758 31024 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:45.991835 31024 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15f8ee4196e415289afe75a34559579, State: Initialized, Role: FOLLOWER
I20260927 00:37:45.991910 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:45.984068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35341 (local address 127.0.0.1:38231)
0927 00:37:45.984469 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:45.984483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:45.984516 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:45.985218 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:45.985235 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:45.985653 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:45.986028 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:45.986450 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:45.987448 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:45.988653 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:45.989122 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:45.989241 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:45.990091 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:45.990096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:45.990171 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:45.990545 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:45.990619 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:45.990641 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:45.990718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:45.990879 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 00:37:45.991098 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260927 00:37:45.992323 31024 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 00:37:45.992470 31024 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:45.992554 31024 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:45.992661 31024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:45.993525 31024 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 00:37:45.993877 31024 leader_election.cc:302] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15f8ee4196e415289afe75a34559579; no voters:
I20260927 00:37:45.994233 31024 leader_election.cc:288] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:45.994292 31029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:45.994493 31029 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [term 1 LEADER]: Becoming Leader. State: Replica: e15f8ee4196e415289afe75a34559579, State: Running, Role: LEADER
I20260927 00:37:45.994742 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:45.994824 31029 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 00:37:45.995457 31024 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:45.996359 31031 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15f8ee4196e415289afe75a34559579. Latest consensus state: current_term: 1 leader_uuid: "e15f8ee4196e415289afe75a34559579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260927 00:37:45.996500 31031 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:45.996898 31030 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15f8ee4196e415289afe75a34559579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15f8ee4196e415289afe75a34559579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260927 00:37:45.997083 31030 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579 [sys.catalog]: This master's current role is: LEADER
W20260927 00:37:46.000097 31049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:37:46.000190 31049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:37:46.000267 31042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:46.000996 31042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:46.004642 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:45.996920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56167 (local address 127.0.0.1:38231)
0927 00:37:45.997012 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:45.997016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:45.997499 (+ 483us) server_negotiation.cc:407] Connection header received
0927 00:37:45.997764 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:45.997769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:45.997821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:45.997939 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:46.000608 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:46.001539 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:46.002777 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:46.003157 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:46.003282 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:46.004201 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:46.004205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:46.004209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:46.004222 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:46.004251 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:46.004266 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:46.004347 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:46.004455 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 00:37:46.004509 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:37:46.006664 31042 catalog_manager.cc:1347] Generated new cluster ID: 18bbde2d6c674509ae2788f6fbdc8d15
I20260927 00:37:46.006721 31042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:46.019265 31042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:46.020251 31042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:46.029896 31042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15f8ee4196e415289afe75a34559579: Generated new TSK 0
I20260927 00:37:46.030581 31042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:46.043920 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 30956
2026-09-27T00:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:46 dist-test-slave-6gxc krb5kdc[31055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:46 dist-test-slave-6gxc krb5kdc[31055](info): set up 2 sockets
Sep 27 00:37:46 dist-test-slave-6gxc krb5kdc[31055](info): commencing operation
krb5kdc: starting...
W20260927 00:37:47.074277 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:47 dist-test-slave-6gxc krb5kdc[31055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:47.196738 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:47.301640 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:47.301914 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:47.301987 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:47.305861 31068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:47.305949 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:47.305992 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:47.306049 31068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:47.306090 31068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:47.311095 31068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:47.312048 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:47.312983 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:47.321336 31073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:47.321345 31076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:47.321571 31068 server_base.cc:1034] running on GCE node
W20260927 00:37:47.321345 31074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:47.322265 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:47.323294 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:47.324476 31068 hybrid_clock.cc:648] HybridClock initialized: now 1790469467324451 us; error 46 us; skew 500 ppm
Sep 27 00:37:47 dist-test-slave-6gxc krb5kdc[31055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:47.327167 31068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:47.327996 31068 webserver.cc:460] Webserver started at http://127.0.0.1:35471/ using document root <none> and password file <none>
I20260927 00:37:47.328536 31068 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:47.328622 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:47.328837 31068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:47.330806 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "894c951651e64fe5a832e1cf4cf5090f"
format_stamp: "Formatted at 2026-09-27 00:37:47 on dist-test-slave-6gxc"
server_key: "89ed2ab2e965ffddaf2e69f7e6a9908e"
server_key_iv: "aaabef475e1b0970a6c2db4004be0d8b"
server_key_version: "encryptionkey@0"
I20260927 00:37:47.331337 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "894c951651e64fe5a832e1cf4cf5090f"
format_stamp: "Formatted at 2026-09-27 00:37:47 on dist-test-slave-6gxc"
server_key: "89ed2ab2e965ffddaf2e69f7e6a9908e"
server_key_iv: "aaabef475e1b0970a6c2db4004be0d8b"
server_key_version: "encryptionkey@0"
I20260927 00:37:47.334981 31068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:37:47.337244 31083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:47.338510 31068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:37:47.338651 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "894c951651e64fe5a832e1cf4cf5090f"
format_stamp: "Formatted at 2026-09-27 00:37:47 on dist-test-slave-6gxc"
server_key: "89ed2ab2e965ffddaf2e69f7e6a9908e"
server_key_iv: "aaabef475e1b0970a6c2db4004be0d8b"
server_key_version: "encryptionkey@0"
I20260927 00:37:47.338753 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:47.343124 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:47.346239 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:47.346419 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:47.353190 31068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39221
I20260927 00:37:47.353204 31135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39221 every 8 connection(s)
I20260927 00:37:47.354030 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:47.356760 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:47.361980 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31068
I20260927 00:37:47.362149 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:47.362187 31136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f: Bootstrap starting.
I20260927 00:37:47.362476 10287 external_mini_cluster.cc:1428] Setting key a3c70098c34fd5f7850443ddcc83baa4
I20260927 00:37:47.364795 31136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:47.365635 31136 log.cc:824] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:47.367467 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:47.363693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43553 (local address 127.0.0.1:39221)
0927 00:37:47.364088 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:47.364102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:47.364136 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:37:47.364750 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:47.364767 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:47.365082 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:47.365347 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:47.366248 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:47.366256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:47.366274 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:47.366289 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:47.366353 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:47.366385 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:47.366501 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:47.366645 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:37:47.366807 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":132,"threads_started":1}
I20260927 00:37:47.367717 31136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f: No bootstrap required, opened a new log
I20260927 00:37:47.370237 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:47.372298 31136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } }
I20260927 00:37:47.372586 31136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:47.372699 31136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894c951651e64fe5a832e1cf4cf5090f, State: Initialized, Role: FOLLOWER
I20260927 00:37:47.373211 31136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } }
I20260927 00:37:47.373425 31136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:47.373503 31136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:47.373632 31136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:47.374336 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:47.372580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56087 (local address 127.0.0.1:39221)
0927 00:37:47.372685 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:47.372689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:47.372884 (+ 195us) server_negotiation.cc:407] Connection header received
0927 00:37:47.373025 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:47.373028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:47.373064 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:47.373126 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:47.373908 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:47.373911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:47.373913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:47.373923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:47.373947 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:47.373957 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:47.374040 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:47.374190 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:37:47.374219 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:37:47.374863 31136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } }
I20260927 00:37:47.375288 31136 leader_election.cc:302] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894c951651e64fe5a832e1cf4cf5090f; no voters:
I20260927 00:37:47.375598 31136 leader_election.cc:288] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:47.375705 31146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:47.375913 31146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [term 1 LEADER]: Becoming Leader. State: Replica: 894c951651e64fe5a832e1cf4cf5090f, State: Running, Role: LEADER
I20260927 00:37:47.376264 31146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } }
I20260927 00:37:47.376785 31136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:47.378049 31147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894c951651e64fe5a832e1cf4cf5090f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } } }
I20260927 00:37:47.378134 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894c951651e64fe5a832e1cf4cf5090f. Latest consensus state: current_term: 1 leader_uuid: "894c951651e64fe5a832e1cf4cf5090f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c951651e64fe5a832e1cf4cf5090f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39221 } } }
I20260927 00:37:47.378196 31147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:47.378229 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:47.378579 31155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:47.381552 31155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:47.386832 31155 catalog_manager.cc:1347] Generated new cluster ID: 369730532b6d43b6ab0c58a6dba84ba7
I20260927 00:37:47.386890 31155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:47.393702 31155 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:47.393983 31155 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:47.410754 31155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894c951651e64fe5a832e1cf4cf5090f: Generated new TSK 0
I20260927 00:37:47.411439 31155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:47.414682 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31068
2026-09-27T00:37:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:47 dist-test-slave-6gxc krb5kdc[31167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:47 dist-test-slave-6gxc krb5kdc[31167](info): set up 2 sockets
Sep 27 00:37:47 dist-test-slave-6gxc krb5kdc[31167](info): commencing operation
krb5kdc: starting...
W20260927 00:37:48.446237 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:48.560541 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:48.667659 31180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:48.667896 31180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:48.667939 31180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:48.671823 31180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:48.671871 31180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:48.671891 31180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:48.671906 31180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:48.671921 31180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:48.676847 31180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:48.677696 31180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:48.678639 31180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:48.686774 31188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:48.686823 31180 server_base.cc:1034] running on GCE node
W20260927 00:37:48.686789 31185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:48.686782 31186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:48.687511 31180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:48.688515 31180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:48.689702 31180 hybrid_clock.cc:648] HybridClock initialized: now 1790469468689673 us; error 40 us; skew 500 ppm
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:48.692387 31180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:48.693219 31180 webserver.cc:460] Webserver started at http://127.0.0.1:42115/ using document root <none> and password file <none>
I20260927 00:37:48.693727 31180 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:48.693812 31180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:48.694036 31180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:48.695936 31180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "52c771ec8d6045dcbd261123ac721b02"
format_stamp: "Formatted at 2026-09-27 00:37:48 on dist-test-slave-6gxc"
server_key: "3b610c29ef5ed5a170d16d052a4062b5"
server_key_iv: "d53b68c7165d421eabec7deb187634cc"
server_key_version: "encryptionkey@0"
I20260927 00:37:48.696466 31180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "52c771ec8d6045dcbd261123ac721b02"
format_stamp: "Formatted at 2026-09-27 00:37:48 on dist-test-slave-6gxc"
server_key: "3b610c29ef5ed5a170d16d052a4062b5"
server_key_iv: "d53b68c7165d421eabec7deb187634cc"
server_key_version: "encryptionkey@0"
I20260927 00:37:48.699818 31180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:37:48.702059 31195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:48.703238 31180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:48.703368 31180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "52c771ec8d6045dcbd261123ac721b02"
format_stamp: "Formatted at 2026-09-27 00:37:48 on dist-test-slave-6gxc"
server_key: "3b610c29ef5ed5a170d16d052a4062b5"
server_key_iv: "d53b68c7165d421eabec7deb187634cc"
server_key_version: "encryptionkey@0"
I20260927 00:37:48.703470 31180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:48.716940 31180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:48.719548 31180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:48.719732 31180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:48.726505 31180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35247
I20260927 00:37:48.726531 31247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35247 every 8 connection(s)
I20260927 00:37:48.727329 31180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:48.729796 31248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:48.734867 31248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: Bootstrap starting.
I20260927 00:37:48.735853 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31180
I20260927 00:37:48.735996 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:48.736274 10287 external_mini_cluster.cc:1428] Setting key 114b2603c574ff8b5afb472f006a489f
I20260927 00:37:48.737548 31248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:48.738312 31248 log.cc:824] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:48.740211 31248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: No bootstrap required, opened a new log
I20260927 00:37:48.743451 31248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260927 00:37:48.743675 31248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:48.743754 31248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c771ec8d6045dcbd261123ac721b02, State: Initialized, Role: FOLLOWER
I20260927 00:37:48.744236 31248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:37:48.744390 31248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:48.744467 31248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:48.744573 31248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:48.745425 31248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260927 00:37:48.745787 31248 leader_election.cc:302] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c771ec8d6045dcbd261123ac721b02; no voters:
I20260927 00:37:48.746146 31248 leader_election.cc:288] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:48.746274 31253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:48.746495 31253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [term 1 LEADER]: Becoming Leader. State: Replica: 52c771ec8d6045dcbd261123ac721b02, State: Running, Role: LEADER
I20260927 00:37:48.746829 31253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260927 00:37:48.747428 31248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:48.748608 31254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c771ec8d6045dcbd261123ac721b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } } }
I20260927 00:37:48.748723 31254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:48.748610 31255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c771ec8d6045dcbd261123ac721b02. Latest consensus state: current_term: 1 leader_uuid: "52c771ec8d6045dcbd261123ac721b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c771ec8d6045dcbd261123ac721b02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } } }
I20260927 00:37:48.749056 31255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:48.750973 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:48.737282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:39455 (local address 127.0.0.1:35247)
0927 00:37:48.737719 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:48.737733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:48.737765 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:37:48.738466 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:48.738483 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:48.738795 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:48.739099 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:48.739883 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:48.740869 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:48.741715 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:48.742235 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:48.742397 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:48.745048 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:48.745053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:48.745064 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:48.745087 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:48.747143 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:48.747698 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:48.747704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:48.747710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:48.747847 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:48.748335 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:48.748338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:48.748341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:48.748832 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:48.749847 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:48.750030 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 00:37:48.750248 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
W20260927 00:37:48.752223 31268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:37:48.752295 31268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:37:48.752368 31262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:48.753175 31262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:48.753754 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:48.761757 31262 catalog_manager.cc:1347] Generated new cluster ID: d9c95ad228444d5f934563576176b5b0
I20260927 00:37:48.761818 31262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:48.768920 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:48.755769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:41847 (local address 127.0.0.1:35247)
0927 00:37:48.755860 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:48.755864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:48.756104 (+ 240us) server_negotiation.cc:407] Connection header received
0927 00:37:48.756195 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:48.756198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:48.756240 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:48.756324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:48.757109 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:48.758290 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:48.759093 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:48.759447 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:48.759558 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:48.761110 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:48.761113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:48.761116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:48.761141 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:48.767163 (+ 6022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:48.767655 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:48.767661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:48.767667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:48.767737 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:48.768066 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:48.768069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:48.768071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:48.768246 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:48.768399 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:48.768657 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:37:48.768745 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:37:48.775411 31262 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:48.776655 31262 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:48.784768 31262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c771ec8d6045dcbd261123ac721b02: Generated new TSK 0
I20260927 00:37:48.785482 31262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:48.789382 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31180
W20260927 00:37:48.794739 31165 connection.cc:535] client connection to 127.0.0.1:35247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35247): Connection reset by peer (error 104)
2026-09-27T00:37:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31279](info): set up 2 sockets
Sep 27 00:37:48 dist-test-slave-6gxc krb5kdc[31279](info): commencing operation
krb5kdc: starting...
W20260927 00:37:49.834234 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:49 dist-test-slave-6gxc krb5kdc[31279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:49.946978 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:50.053236 31292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:50.053467 31292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:50.053508 31292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:50.057370 31292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:50.057421 31292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:50.057446 31292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:50.057463 31292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:50.057500 31292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:50.062568 31292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:50.063457 31292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:50.064366 31292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:50.072351 31298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:50.072402 31300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:50.072403 31297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:50.072580 31292 server_base.cc:1034] running on GCE node
I20260927 00:37:50.073137 31292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:50.074172 31292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:50.075328 31292 hybrid_clock.cc:648] HybridClock initialized: now 1790469470075304 us; error 36 us; skew 500 ppm
Sep 27 00:37:50 dist-test-slave-6gxc krb5kdc[31279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:50.081975 31292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:50.082810 31292 webserver.cc:460] Webserver started at http://127.0.0.1:34001/ using document root <none> and password file <none>
I20260927 00:37:50.083295 31292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:50.083356 31292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:50.083526 31292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:50.085457 31292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "cd15122dd2e1433ba6b530f7df1b00bc"
format_stamp: "Formatted at 2026-09-27 00:37:50 on dist-test-slave-6gxc"
server_key: "1fa22ae68c44231141be490473065f9a"
server_key_iv: "0333d05040e229026aeac7f07f4cf42a"
server_key_version: "encryptionkey@0"
I20260927 00:37:50.085973 31292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "cd15122dd2e1433ba6b530f7df1b00bc"
format_stamp: "Formatted at 2026-09-27 00:37:50 on dist-test-slave-6gxc"
server_key: "1fa22ae68c44231141be490473065f9a"
server_key_iv: "0333d05040e229026aeac7f07f4cf42a"
server_key_version: "encryptionkey@0"
I20260927 00:37:50.089622 31292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:37:50.091899 31307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:50.093050 31292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:37:50.093148 31292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "cd15122dd2e1433ba6b530f7df1b00bc"
format_stamp: "Formatted at 2026-09-27 00:37:50 on dist-test-slave-6gxc"
server_key: "1fa22ae68c44231141be490473065f9a"
server_key_iv: "0333d05040e229026aeac7f07f4cf42a"
server_key_version: "encryptionkey@0"
I20260927 00:37:50.093225 31292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:50.101398 31292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:50.104305 31292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:50.104447 31292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:50.111338 31292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33091
I20260927 00:37:50.111351 31359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33091 every 8 connection(s)
I20260927 00:37:50.112188 31292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:50.114740 31360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:50.119609 31360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc: Bootstrap starting.
I20260927 00:37:50.121752 31360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:50.122355 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31292
I20260927 00:37:50.122514 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:50.122586 31360 log.cc:824] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:50.122807 10287 external_mini_cluster.cc:1428] Setting key 358800cca66e093b6b94632e592c75b0
I20260927 00:37:50.124483 31360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc: No bootstrap required, opened a new log
I20260927 00:37:50.128280 31360 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260927 00:37:50.128473 31360 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:50.128530 31360 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd15122dd2e1433ba6b530f7df1b00bc, State: Initialized, Role: FOLLOWER
I20260927 00:37:50.128970 31360 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260927 00:37:50.129134 31360 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:50.129213 31360 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:50.129328 31360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:50.130290 31360 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260927 00:37:50.130601 31360 leader_election.cc:302] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd15122dd2e1433ba6b530f7df1b00bc; no voters:
I20260927 00:37:50.130928 31360 leader_election.cc:288] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:50.130993 31365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:50.131196 31365 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [term 1 LEADER]: Becoming Leader. State: Replica: cd15122dd2e1433ba6b530f7df1b00bc, State: Running, Role: LEADER
I20260927 00:37:50.131546 31365 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260927 00:37:50.131791 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:50.123848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:51961 (local address 127.0.0.1:33091)
0927 00:37:50.124289 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:50.124309 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:50.124352 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:37:50.125009 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:50.125026 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:50.125329 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:50.125600 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:50.125918 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:50.126913 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:50.128147 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:50.128720 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:50.128898 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:50.129812 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:50.129817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:50.129828 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:50.130253 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:50.130305 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:50.130327 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:50.130545 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:50.130693 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:37:50.130822 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
I20260927 00:37:50.132030 31360 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:50.132714 31365 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd15122dd2e1433ba6b530f7df1b00bc. Latest consensus state: current_term: 1 leader_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260927 00:37:50.132814 31365 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:50.133368 31366 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd15122dd2e1433ba6b530f7df1b00bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260927 00:37:50.133463 31366 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:50.134516 31374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:50.135463 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:50.137743 31374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:50.143183 31374 catalog_manager.cc:1347] Generated new cluster ID: 6f1af81b9aac4838952524df566bcb56
I20260927 00:37:50.143428 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:50.138491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:53205 (local address 127.0.0.1:33091)
0927 00:37:50.138600 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:50.138604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:50.139144 (+ 540us) server_negotiation.cc:407] Connection header received
0927 00:37:50.139258 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:50.139262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:50.139303 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:50.139402 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:50.139662 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:50.140440 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:50.141495 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:50.141771 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:50.141867 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:50.142861 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:50.142865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:50.142867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:50.142879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:50.142905 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:50.142917 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:50.142974 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:50.143224 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:37:50.143259 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:37:50.143297 31374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:50.161897 31374 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:50.162897 31374 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:50.167936 31374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd15122dd2e1433ba6b530f7df1b00bc: Generated new TSK 0
I20260927 00:37:50.168581 31374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:50.175365 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31292
W20260927 00:37:50.180496 31277 connection.cc:535] client connection to 127.0.0.1:33091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33091): Connection reset by peer (error 104)
2026-09-27T00:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:50 dist-test-slave-6gxc krb5kdc[31391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:50 dist-test-slave-6gxc krb5kdc[31391](info): set up 2 sockets
Sep 27 00:37:50 dist-test-slave-6gxc krb5kdc[31391](info): commencing operation
krb5kdc: starting...
W20260927 00:37:51.202304 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:51 dist-test-slave-6gxc krb5kdc[31391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:51.317227 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:51.423677 31404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:51.423998 31404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:51.424084 31404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:51.428014 31404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:51.428102 31404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:51.428164 31404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:51.428238 31404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:51.428279 31404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:51.433310 31404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:51.434274 31404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:51.435213 31404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:51.443331 31410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:51.443377 31412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:51.443446 31404 server_base.cc:1034] running on GCE node
W20260927 00:37:51.443672 31409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:51.444207 31404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:51.445196 31404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:51.446350 31404 hybrid_clock.cc:648] HybridClock initialized: now 1790469471446333 us; error 29 us; skew 500 ppm
Sep 27 00:37:51 dist-test-slave-6gxc krb5kdc[31391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:51.448823 31404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:51.449563 31404 webserver.cc:460] Webserver started at http://127.0.0.1:34527/ using document root <none> and password file <none>
I20260927 00:37:51.450033 31404 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:51.450150 31404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:51.450327 31404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:51.452198 31404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "8078f2ecde0e4c169759883437a0f8fd"
format_stamp: "Formatted at 2026-09-27 00:37:51 on dist-test-slave-6gxc"
server_key: "dd980932e3bbe70a6ad4826979fd91b5"
server_key_iv: "92e3c4b09e69417404c36e51620d8e3d"
server_key_version: "encryptionkey@0"
I20260927 00:37:51.452665 31404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "8078f2ecde0e4c169759883437a0f8fd"
format_stamp: "Formatted at 2026-09-27 00:37:51 on dist-test-slave-6gxc"
server_key: "dd980932e3bbe70a6ad4826979fd91b5"
server_key_iv: "92e3c4b09e69417404c36e51620d8e3d"
server_key_version: "encryptionkey@0"
I20260927 00:37:51.456082 31404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:37:51.458221 31419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:51.459342 31404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:51.459465 31404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "8078f2ecde0e4c169759883437a0f8fd"
format_stamp: "Formatted at 2026-09-27 00:37:51 on dist-test-slave-6gxc"
server_key: "dd980932e3bbe70a6ad4826979fd91b5"
server_key_iv: "92e3c4b09e69417404c36e51620d8e3d"
server_key_version: "encryptionkey@0"
I20260927 00:37:51.459561 31404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:51.464377 31404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:51.467022 31404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:51.467206 31404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:51.473500 31404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41249
I20260927 00:37:51.473559 31471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41249 every 8 connection(s)
I20260927 00:37:51.474403 31404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:51.476843 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:51.481721 31472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd: Bootstrap starting.
I20260927 00:37:51.482354 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31404
I20260927 00:37:51.482509 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:51.482788 10287 external_mini_cluster.cc:1428] Setting key f7b22318c991cd2040fea84353d7bb9f
I20260927 00:37:51.484345 31472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:51.485018 31472 log.cc:824] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:51.487138 31472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd: No bootstrap required, opened a new log
I20260927 00:37:51.487738 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:51.484012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:35403 (local address 127.0.0.1:41249)
0927 00:37:51.484419 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:51.484440 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:51.484483 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:37:51.485060 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:51.485109 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:51.485433 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:51.485774 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:51.486585 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:51.486589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:51.486599 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:51.486608 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:51.486654 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:51.486671 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:51.486751 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:51.486889 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:37:51.487050 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":121,"threads_started":1}
I20260927 00:37:51.490437 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:51.491014 31472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260927 00:37:51.491204 31472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:51.491259 31472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8078f2ecde0e4c169759883437a0f8fd, State: Initialized, Role: FOLLOWER
I20260927 00:37:51.491719 31472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260927 00:37:51.491889 31472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:51.491963 31472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:51.492053 31472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:51.493126 31472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260927 00:37:51.493489 31472 leader_election.cc:302] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8078f2ecde0e4c169759883437a0f8fd; no voters:
I20260927 00:37:51.493805 31472 leader_election.cc:288] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:51.494022 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:51.492504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:41577 (local address 127.0.0.1:41249)
0927 00:37:51.492608 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:51.492613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:51.492741 (+ 128us) server_negotiation.cc:407] Connection header received
0927 00:37:51.492868 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:51.492870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:51.492893 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:51.492953 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:51.493650 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:51.493655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:51.493658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:51.493670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:51.493693 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:51.493704 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:51.493800 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:51.493868 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:37:51.493906 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:37:51.493891 31482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:51.494271 31482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [term 1 LEADER]: Becoming Leader. State: Replica: 8078f2ecde0e4c169759883437a0f8fd, State: Running, Role: LEADER
I20260927 00:37:51.494607 31482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260927 00:37:51.495075 31472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:51.495688 31482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8078f2ecde0e4c169759883437a0f8fd. Latest consensus state: current_term: 1 leader_uuid: "8078f2ecde0e4c169759883437a0f8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } } }
I20260927 00:37:51.495690 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8078f2ecde0e4c169759883437a0f8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078f2ecde0e4c169759883437a0f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } } }
I20260927 00:37:51.495841 31482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:51.495841 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:51.496235 31491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:51.499261 31491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:51.504220 31491 catalog_manager.cc:1347] Generated new cluster ID: 0bb384bf3e26480fa6727ba22aca2f19
I20260927 00:37:51.504292 31491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:51.528254 31491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:51.528544 31491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:51.537777 31491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8078f2ecde0e4c169759883437a0f8fd: Generated new TSK 0
I20260927 00:37:51.538405 31491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:51.543107 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31404
2026-09-27T00:37:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:51 dist-test-slave-6gxc krb5kdc[31503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:51 dist-test-slave-6gxc krb5kdc[31503](info): set up 2 sockets
Sep 27 00:37:51 dist-test-slave-6gxc krb5kdc[31503](info): commencing operation
krb5kdc: starting...
W20260927 00:37:52.566251 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:52 dist-test-slave-6gxc krb5kdc[31503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:52.683041 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:52.786679 31516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:52.786932 31516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:52.787002 31516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:52.790951 31516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:52.791023 31516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:52.791062 31516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:52.791116 31516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:52.791157 31516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:52.796087 31516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:52.796973 31516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:52.797864 31516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:52.805888 31524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:52.805861 31522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:52.805850 31521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:52.805902 31516 server_base.cc:1034] running on GCE node
I20260927 00:37:52.806593 31516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:52.807543 31516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:52.808696 31516 hybrid_clock.cc:648] HybridClock initialized: now 1790469472808671 us; error 38 us; skew 500 ppm
Sep 27 00:37:52 dist-test-slave-6gxc krb5kdc[31503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:52.811352 31516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:52.812176 31516 webserver.cc:460] Webserver started at http://127.0.0.1:43173/ using document root <none> and password file <none>
I20260927 00:37:52.812668 31516 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:52.812726 31516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:52.812915 31516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:52.814810 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "3d98fe80c63f405daedb27ba14b2395d"
format_stamp: "Formatted at 2026-09-27 00:37:52 on dist-test-slave-6gxc"
server_key: "a715144271900d00ccd00dc4f8ce498a"
server_key_iv: "88ee19e48a9833efc9a556154f904f6d"
server_key_version: "encryptionkey@0"
I20260927 00:37:52.815336 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "3d98fe80c63f405daedb27ba14b2395d"
format_stamp: "Formatted at 2026-09-27 00:37:52 on dist-test-slave-6gxc"
server_key: "a715144271900d00ccd00dc4f8ce498a"
server_key_iv: "88ee19e48a9833efc9a556154f904f6d"
server_key_version: "encryptionkey@0"
I20260927 00:37:52.818634 31516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:52.820771 31531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:52.821857 31516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:52.821997 31516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "3d98fe80c63f405daedb27ba14b2395d"
format_stamp: "Formatted at 2026-09-27 00:37:52 on dist-test-slave-6gxc"
server_key: "a715144271900d00ccd00dc4f8ce498a"
server_key_iv: "88ee19e48a9833efc9a556154f904f6d"
server_key_version: "encryptionkey@0"
I20260927 00:37:52.822151 31516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:52.840150 31516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:52.842801 31516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:52.842952 31516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:52.849803 31516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34875
I20260927 00:37:52.849859 31583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34875 every 8 connection(s)
I20260927 00:37:52.850730 31516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:52.853245 31584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:52.858234 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31516
I20260927 00:37:52.858390 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:52.858388 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d: Bootstrap starting.
I20260927 00:37:52.858721 10287 external_mini_cluster.cc:1428] Setting key 8d3f3e685bba272ae6fa27eed2e463a0
I20260927 00:37:52.860611 31584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:52.861265 31584 log.cc:824] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:52.863272 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d: No bootstrap required, opened a new log
Sep 27 00:37:52 dist-test-slave-6gxc krb5kdc[31503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:37:52.867619 31584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260927 00:37:52.867841 31584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:52.867921 31584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d98fe80c63f405daedb27ba14b2395d, State: Initialized, Role: FOLLOWER
I20260927 00:37:52.868422 31584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260927 00:37:52.868592 31584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:52.868662 31584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:52.868765 31584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:52.869536 31584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260927 00:37:52.869860 31584 leader_election.cc:302] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d98fe80c63f405daedb27ba14b2395d; no voters:
I20260927 00:37:52.870198 31584 leader_election.cc:288] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:52.870255 31589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:52.870463 31589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [term 1 LEADER]: Becoming Leader. State: Replica: 3d98fe80c63f405daedb27ba14b2395d, State: Running, Role: LEADER
I20260927 00:37:52.870738 31589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260927 00:37:52.871373 31584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:52.871866 31590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d98fe80c63f405daedb27ba14b2395d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260927 00:37:52.871894 31591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d98fe80c63f405daedb27ba14b2395d. Latest consensus state: current_term: 1 leader_uuid: "3d98fe80c63f405daedb27ba14b2395d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d98fe80c63f405daedb27ba14b2395d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260927 00:37:52.872016 31590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:52.872020 31591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:52.872457 31593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:52.874302 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:52.859983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:33323 (local address 127.0.0.1:34875)
0927 00:37:52.860361 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:52.860378 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:52.860424 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:37:52.861038 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:52.861054 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:52.861366 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:52.861765 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:52.862544 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:52.863565 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:52.864712 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:52.865169 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:52.865285 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:52.868193 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:52.868199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:52.868213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:52.868241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:52.870246 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:52.870815 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:52.870823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:52.870830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:52.871020 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:52.871428 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:52.871431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:52.871433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:52.871921 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:52.872941 (+ 1020us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:52.873122 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:37:52.873336 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":130,"threads_started":1}
I20260927 00:37:52.875854 31593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:52.877408 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:52.881713 31593 catalog_manager.cc:1347] Generated new cluster ID: c54dc22cdf584014972c33cb7001c57d
I20260927 00:37:52.881771 31593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:52.887688 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:52.879443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:44775 (local address 127.0.0.1:34875)
0927 00:37:52.879532 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:52.879537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:52.879908 (+ 371us) server_negotiation.cc:407] Connection header received
0927 00:37:52.880024 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:52.880027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:52.880071 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:52.880144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:52.880992 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:52.881688 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:52.882647 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:52.882906 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:52.883015 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:52.884384 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:52.884388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:52.884390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:52.884422 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:52.885827 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:52.886371 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:52.886374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:52.886376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:52.886456 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:52.886779 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:52.886782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:52.886783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:52.886942 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:52.887093 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:52.887354 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 00:37:52.887450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 00:37:52.934258 31593 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:52.935343 31593 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:52.942689 31593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d98fe80c63f405daedb27ba14b2395d: Generated new TSK 0
I20260927 00:37:52.943354 31593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:52.947485 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31516
W20260927 00:37:52.952659 31501 connection.cc:535] client connection to 127.0.0.1:34875 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34875): Connection reset by peer (error 104)
2026-09-27T00:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:53 dist-test-slave-6gxc krb5kdc[31615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:53 dist-test-slave-6gxc krb5kdc[31615](info): set up 2 sockets
Sep 27 00:37:53 dist-test-slave-6gxc krb5kdc[31615](info): commencing operation
krb5kdc: starting...
W20260927 00:37:53.994172 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:54 dist-test-slave-6gxc krb5kdc[31615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:54.106024 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:54.209930 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:54.210248 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:54.210294 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:54.214105 31628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:54.214169 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:54.214186 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:54.214202 31628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:54.214217 31628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:54.219043 31628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:54.219844 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:54.220737 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:37:54.228818 31628 server_base.cc:1034] running on GCE node
W20260927 00:37:54.228783 31634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:54.228760 31633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:54.228785 31636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:54.229470 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:54.230587 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:54.231752 31628 hybrid_clock.cc:648] HybridClock initialized: now 1790469474231735 us; error 30 us; skew 500 ppm
Sep 27 00:37:54 dist-test-slave-6gxc krb5kdc[31615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:54.234299 31628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:54.235123 31628 webserver.cc:460] Webserver started at http://127.0.0.1:35071/ using document root <none> and password file <none>
I20260927 00:37:54.235642 31628 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:54.235730 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:54.235948 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:54.237797 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "c0513667daa642f6b91ad67d60abcde9"
format_stamp: "Formatted at 2026-09-27 00:37:54 on dist-test-slave-6gxc"
server_key: "c5ab30aa9139defd7500d3caa00a6619"
server_key_iv: "21749e04a14fbba664a920c23a2d4b4a"
server_key_version: "encryptionkey@0"
I20260927 00:37:54.238343 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "c0513667daa642f6b91ad67d60abcde9"
format_stamp: "Formatted at 2026-09-27 00:37:54 on dist-test-slave-6gxc"
server_key: "c5ab30aa9139defd7500d3caa00a6619"
server_key_iv: "21749e04a14fbba664a920c23a2d4b4a"
server_key_version: "encryptionkey@0"
I20260927 00:37:54.241712 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:37:54.243988 31643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:54.245054 31628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:54.245193 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "c0513667daa642f6b91ad67d60abcde9"
format_stamp: "Formatted at 2026-09-27 00:37:54 on dist-test-slave-6gxc"
server_key: "c5ab30aa9139defd7500d3caa00a6619"
server_key_iv: "21749e04a14fbba664a920c23a2d4b4a"
server_key_version: "encryptionkey@0"
I20260927 00:37:54.245296 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:54.259344 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:54.262014 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:54.262238 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:54.268831 31628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35037
I20260927 00:37:54.268859 31695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35037 every 8 connection(s)
I20260927 00:37:54.269658 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:54.271044 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31628
I20260927 00:37:54.271144 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:54.271432 10287 external_mini_cluster.cc:1428] Setting key ef811a80bb13f4d75f2af9e08a204c33
I20260927 00:37:54.272313 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:54.277432 31696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9: Bootstrap starting.
I20260927 00:37:54.279575 31696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:54.279670 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:54.272719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:39017 (local address 127.0.0.1:35037)
0927 00:37:54.273119 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:54.273136 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:54.273167 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:37:54.273757 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:54.273772 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:54.274072 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:54.274447 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:54.274833 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:54.275762 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:54.276675 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:54.277118 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:54.277289 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:54.278180 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:54.278188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:54.278204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:54.278542 (+ 338us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:54.278589 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:54.278610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:54.278761 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:54.278907 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 00:37:54.279035 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260927 00:37:54.280323 31696 log.cc:824] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:54.282280 31696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9: No bootstrap required, opened a new log
I20260927 00:37:54.282475 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:54.286193 31696 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260927 00:37:54.286433 31696 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:54.286499 31696 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0513667daa642f6b91ad67d60abcde9, State: Initialized, Role: FOLLOWER
I20260927 00:37:54.286893 31696 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260927 00:37:54.287052 31696 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:54.287161 31696 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:54.287273 31696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:54.288295 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:54.284311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:50377 (local address 127.0.0.1:35037)
0927 00:37:54.284433 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:54.284438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:54.284616 (+ 178us) server_negotiation.cc:407] Connection header received
0927 00:37:54.284720 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:54.284722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:54.284756 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:54.284818 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:54.285098 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:54.285551 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:54.286648 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:54.286926 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:54.287006 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:54.287820 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:54.287822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:54.287824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:54.287834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:54.287855 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:54.287865 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:54.287932 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:54.288097 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 00:37:54.288142 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:37:54.288389 31696 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260927 00:37:54.288735 31696 leader_election.cc:302] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0513667daa642f6b91ad67d60abcde9; no voters:
I20260927 00:37:54.289055 31696 leader_election.cc:288] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:54.289120 31706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:54.289300 31706 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [term 1 LEADER]: Becoming Leader. State: Replica: c0513667daa642f6b91ad67d60abcde9, State: Running, Role: LEADER
I20260927 00:37:54.289613 31706 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260927 00:37:54.290237 31696 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:54.291378 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0513667daa642f6b91ad67d60abcde9. Latest consensus state: current_term: 1 leader_uuid: "c0513667daa642f6b91ad67d60abcde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } } }
I20260927 00:37:54.291424 31707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0513667daa642f6b91ad67d60abcde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0513667daa642f6b91ad67d60abcde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } } }
I20260927 00:37:54.291489 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:54.291515 31707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:54.291908 31715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:54.294559 31715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:54.299211 31715 catalog_manager.cc:1347] Generated new cluster ID: f8c206f1343c4ab98964ef8d90744279
I20260927 00:37:54.299273 31715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:54.317461 31715 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:54.318394 31715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:54.327258 31715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0513667daa642f6b91ad67d60abcde9: Generated new TSK 0
I20260927 00:37:54.327889 31715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:54.334609 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31628
W20260927 00:37:54.340082 31613 connection.cc:535] client connection to 127.0.0.1:35037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35037): Connection reset by peer (error 104)
2026-09-27T00:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:54 dist-test-slave-6gxc krb5kdc[31727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:54 dist-test-slave-6gxc krb5kdc[31727](info): set up 2 sockets
Sep 27 00:37:54 dist-test-slave-6gxc krb5kdc[31727](info): commencing operation
krb5kdc: starting...
W20260927 00:37:55.374321 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:55 dist-test-slave-6gxc krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:37:55.486438 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:55.591228 31740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:55.591444 31740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:55.591490 31740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:55.595250 31740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:55.595293 31740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:55.595312 31740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:55.595328 31740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:55.595342 31740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:55.600211 31740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:55.601094 31740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:55.601954 31740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:55.609759 31745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:55.609740 31746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:55.609897 31740 server_base.cc:1034] running on GCE node
W20260927 00:37:55.609743 31748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:55.610596 31740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:55.611644 31740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:55.612846 31740 hybrid_clock.cc:648] HybridClock initialized: now 1790469475612819 us; error 41 us; skew 500 ppm
Sep 27 00:37:55 dist-test-slave-6gxc krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:55.619802 31740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:37:55.620677 31740 webserver.cc:460] Webserver started at http://127.0.0.1:37831/ using document root <none> and password file <none>
I20260927 00:37:55.621232 31740 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:55.621290 31740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:55.621443 31740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:55.623364 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "a5798c4f5a59485ea1904afb02ba7ba5"
format_stamp: "Formatted at 2026-09-27 00:37:55 on dist-test-slave-6gxc"
server_key: "b5ad24d0c637396a84af663b8d838beb"
server_key_iv: "6ed96a12c1d705a2478004a0a0ec17bf"
server_key_version: "encryptionkey@0"
I20260927 00:37:55.623847 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "a5798c4f5a59485ea1904afb02ba7ba5"
format_stamp: "Formatted at 2026-09-27 00:37:55 on dist-test-slave-6gxc"
server_key: "b5ad24d0c637396a84af663b8d838beb"
server_key_iv: "6ed96a12c1d705a2478004a0a0ec17bf"
server_key_version: "encryptionkey@0"
I20260927 00:37:55.627296 31740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:37:55.629520 31755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:55.630722 31740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:55.630841 31740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "a5798c4f5a59485ea1904afb02ba7ba5"
format_stamp: "Formatted at 2026-09-27 00:37:55 on dist-test-slave-6gxc"
server_key: "b5ad24d0c637396a84af663b8d838beb"
server_key_iv: "6ed96a12c1d705a2478004a0a0ec17bf"
server_key_version: "encryptionkey@0"
I20260927 00:37:55.630987 31740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:55.635310 31740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:55.638362 31740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:55.638547 31740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:55.645488 31740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36581
I20260927 00:37:55.645501 31807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36581 every 8 connection(s)
I20260927 00:37:55.646404 31740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:55.648903 31808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:55.651355 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31740
I20260927 00:37:55.651487 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:55.651762 10287 external_mini_cluster.cc:1428] Setting key 9f870efaec1d1340ae854c11a7a9a1c1
I20260927 00:37:55.654503 31808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5: Bootstrap starting.
I20260927 00:37:55.656538 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:55.653048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:59941 (local address 127.0.0.1:36581)
0927 00:37:55.653436 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:55.653448 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:55.653480 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:37:55.654057 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:55.654072 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:55.654384 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:55.654690 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:55.655428 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:55.655432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:55.655442 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:55.655450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:55.655495 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:55.655511 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:55.655612 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:55.655738 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:37:55.655892 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260927 00:37:55.656955 31808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:55.657656 31808 log.cc:824] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:55.659207 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:55.659584 31808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5: No bootstrap required, opened a new log
I20260927 00:37:55.662787 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:55.661169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:38359 (local address 127.0.0.1:36581)
0927 00:37:55.661274 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:55.661277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:55.661490 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:37:55.661608 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:55.661610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:55.661636 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:55.661700 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:55.662414 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:55.662417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:55.662419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:55.662428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:55.662451 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:55.662460 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:55.662528 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:55.662648 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 00:37:55.662678 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:37:55.663930 31808 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } }
I20260927 00:37:55.664129 31808 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:55.664201 31808 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5798c4f5a59485ea1904afb02ba7ba5, State: Initialized, Role: FOLLOWER
I20260927 00:37:55.664574 31808 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } }
I20260927 00:37:55.664708 31808 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:55.664773 31808 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:55.664868 31808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:55.665681 31808 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } }
I20260927 00:37:55.666000 31808 leader_election.cc:302] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5798c4f5a59485ea1904afb02ba7ba5; no voters:
I20260927 00:37:55.666330 31808 leader_election.cc:288] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:55.666435 31818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:55.666642 31818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [term 1 LEADER]: Becoming Leader. State: Replica: a5798c4f5a59485ea1904afb02ba7ba5, State: Running, Role: LEADER
I20260927 00:37:55.666939 31818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } }
I20260927 00:37:55.667419 31808 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:55.668538 31819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } } }
I20260927 00:37:55.668587 31820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5798c4f5a59485ea1904afb02ba7ba5. Latest consensus state: current_term: 1 leader_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5798c4f5a59485ea1904afb02ba7ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36581 } } }
I20260927 00:37:55.668658 31819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:55.668675 31820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:55.669041 31827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:55.671969 31827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:55.676646 31827 catalog_manager.cc:1347] Generated new cluster ID: 46fe0b0674854aeabc62594f12e6a4fe
I20260927 00:37:55.676704 31827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:55.699863 31827 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:55.700140 31827 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:55.711037 31827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5798c4f5a59485ea1904afb02ba7ba5: Generated new TSK 0
I20260927 00:37:55.711624 31827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:55.717808 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31740
2026-09-27T00:37:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:55 dist-test-slave-6gxc krb5kdc[31839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:55 dist-test-slave-6gxc krb5kdc[31839](info): set up 2 sockets
Sep 27 00:37:55 dist-test-slave-6gxc krb5kdc[31839](info): commencing operation
krb5kdc: starting...
W20260927 00:37:56.742491 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:56 dist-test-slave-6gxc krb5kdc[31839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:56.857297 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38533
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:56.961835 31852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:56.962106 31852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:56.962237 31852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:56.966164 31852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:56.966243 31852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:56.966288 31852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:56.966346 31852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:56.966385 31852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:56.971333 31852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:38533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:56.972257 31852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:56.973176 31852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:56.981223 31858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:56.981215 31860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:56.981192 31857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:56.981261 31852 server_base.cc:1034] running on GCE node
I20260927 00:37:56.981950 31852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:56.982973 31852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:56.984163 31852 hybrid_clock.cc:648] HybridClock initialized: now 1790469476984137 us; error 40 us; skew 500 ppm
Sep 27 00:37:56 dist-test-slave-6gxc krb5kdc[31839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:56.986810 31852 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:56.987782 31852 webserver.cc:460] Webserver started at http://127.10.11.254:40661/ using document root <none> and password file <none>
I20260927 00:37:56.988440 31852 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:56.988504 31852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:56.988742 31852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:56.991132 31852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "94822ec614ad4cb6bd42c031a83ad20a"
format_stamp: "Formatted at 2026-09-27 00:37:56 on dist-test-slave-6gxc"
server_key: "2aa91f84d90ba055a72ecbf014b4ba14"
server_key_iv: "fc50a8ddc4b40fd26477c60c42c77268"
server_key_version: "encryptionkey@0"
I20260927 00:37:56.991607 31852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "94822ec614ad4cb6bd42c031a83ad20a"
format_stamp: "Formatted at 2026-09-27 00:37:56 on dist-test-slave-6gxc"
server_key: "2aa91f84d90ba055a72ecbf014b4ba14"
server_key_iv: "fc50a8ddc4b40fd26477c60c42c77268"
server_key_version: "encryptionkey@0"
I20260927 00:37:56.994956 31852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:37:56.997087 31867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:56.998281 31852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:56.998373 31852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "94822ec614ad4cb6bd42c031a83ad20a"
format_stamp: "Formatted at 2026-09-27 00:37:56 on dist-test-slave-6gxc"
server_key: "2aa91f84d90ba055a72ecbf014b4ba14"
server_key_iv: "fc50a8ddc4b40fd26477c60c42c77268"
server_key_version: "encryptionkey@0"
I20260927 00:37:56.998446 31852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:57.008874 31852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:57.011796 31852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:57.011930 31852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:57.018664 31852 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:38533
I20260927 00:37:57.018680 31919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:38533 every 8 connection(s)
I20260927 00:37:57.019497 31852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:57.022027 31920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:57.022378 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31852
I20260927 00:37:57.022518 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:57.022804 10287 external_mini_cluster.cc:1428] Setting key 008335aef3218a7f8d04e1da3e9e903e
I20260927 00:37:57.028540 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a: Bootstrap starting.
I20260927 00:37:57.031176 31920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a: Neither blocks nor log segments found. Creating new log.
Sep 27 00:37:57 dist-test-slave-6gxc krb5kdc[31839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:37:57.031919 31920 log.cc:824] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:57.033596 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a: No bootstrap required, opened a new log
I20260927 00:37:57.036820 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:57.024101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47145 (local address 127.10.11.254:38533)
0927 00:37:57.024470 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:57.024493 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:57.024526 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:57.025156 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:57.025171 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:57.025482 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:57.025737 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:57.026676 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:57.027631 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:57.028802 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:57.029259 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:57.029381 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:57.032336 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:57.032341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:57.032353 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:57.032378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:57.034090 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:57.034640 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:57.034646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:57.034659 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:57.034758 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:57.035098 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:57.035101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:57.035103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:57.035459 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:57.035668 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:57.035952 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:37:57.036154 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260927 00:37:57.037055 31920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } }
I20260927 00:37:57.037279 31920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:57.037353 31920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94822ec614ad4cb6bd42c031a83ad20a, State: Initialized, Role: FOLLOWER
I20260927 00:37:57.037798 31920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } }
I20260927 00:37:57.037958 31920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:57.038019 31920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:57.038130 31920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:57.039153 31920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } }
I20260927 00:37:57.039512 31920 leader_election.cc:302] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94822ec614ad4cb6bd42c031a83ad20a; no voters:
I20260927 00:37:57.039844 31920 leader_election.cc:288] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:57.039870 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:57.039907 31925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:57.040107 31925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [term 1 LEADER]: Becoming Leader. State: Replica: 94822ec614ad4cb6bd42c031a83ad20a, State: Running, Role: LEADER
I20260927 00:37:57.040431 31925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } }
I20260927 00:37:57.041025 31920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:57.041524 31925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94822ec614ad4cb6bd42c031a83ad20a. Latest consensus state: current_term: 1 leader_uuid: "94822ec614ad4cb6bd42c031a83ad20a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } } }
I20260927 00:37:57.041693 31925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:57.041944 31926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94822ec614ad4cb6bd42c031a83ad20a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94822ec614ad4cb6bd42c031a83ad20a" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 38533 } } }
I20260927 00:37:57.042042 31926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:57.042162 31935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:57.045218 31935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:57.051230 31935 catalog_manager.cc:1347] Generated new cluster ID: 795e0b46604d41dcb086b4be71f0429d
I20260927 00:37:57.051334 31935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:57.052145 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:57.041949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39889 (local address 127.10.11.254:38533)
0927 00:37:57.042057 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:57.042062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:57.043338 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 00:37:57.043505 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:57.043508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:57.043555 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:57.043637 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:57.044491 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:57.045038 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:57.046140 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:57.046473 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:57.046581 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:57.048155 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:57.048160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:57.048163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:37:57.048189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:57.050217 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:57.050772 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:57.050776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:57.050778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:57.050862 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:37:57.051243 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:37:57.051246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:37:57.051248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:37:57.051476 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:57.051626 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:57.051887 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 00:37:57.051976 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:37:57.067411 31935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:57.068418 31935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:57.075592 31935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94822ec614ad4cb6bd42c031a83ad20a: Generated new TSK 0
I20260927 00:37:57.076260 31935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:57.088151 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31852
W20260927 00:37:57.093794 31837 connection.cc:535] client connection to 127.10.11.254:38533 recv error: Network error: failed to read from TLS socket (remote: 127.10.11.254:38533): Connection reset by peer (error 104)
2026-09-27T00:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:57 dist-test-slave-6gxc krb5kdc[31951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:57 dist-test-slave-6gxc krb5kdc[31951](info): set up 2 sockets
Sep 27 00:37:57 dist-test-slave-6gxc krb5kdc[31951](info): commencing operation
krb5kdc: starting...
W20260927 00:37:58.122836 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:58 dist-test-slave-6gxc krb5kdc[31951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:58.237185 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41531
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:36017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:58.342366 31964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:58.342593 31964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:58.342633 31964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:58.346446 31964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:58.346493 31964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:58.346511 31964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:58.346529 31964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:58.346544 31964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:58.351500 31964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:36017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:41531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:58.352365 31964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:58.353237 31964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:58.361287 31970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:58.361290 31972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:58.361290 31969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:58.361611 31964 server_base.cc:1034] running on GCE node
I20260927 00:37:58.362206 31964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:58.363176 31964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:58.364336 31964 hybrid_clock.cc:648] HybridClock initialized: now 1790469478364310 us; error 39 us; skew 500 ppm
Sep 27 00:37:58 dist-test-slave-6gxc krb5kdc[31951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:58.366962 31964 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:58.367714 31964 webserver.cc:460] Webserver started at http://127.10.11.254:36055/ using document root <none> and password file <none>
I20260927 00:37:58.368220 31964 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:58.368278 31964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:58.368425 31964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:58.370321 31964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "30aaf91af22a49c88db46c21dda140fd"
format_stamp: "Formatted at 2026-09-27 00:37:58 on dist-test-slave-6gxc"
server_key: "c818c28240bc88b0923020b338f956f1"
server_key_iv: "d91e3d315edb48e3a82af0bcc92cc13b"
server_key_version: "encryptionkey@0"
I20260927 00:37:58.370797 31964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "30aaf91af22a49c88db46c21dda140fd"
format_stamp: "Formatted at 2026-09-27 00:37:58 on dist-test-slave-6gxc"
server_key: "c818c28240bc88b0923020b338f956f1"
server_key_iv: "d91e3d315edb48e3a82af0bcc92cc13b"
server_key_version: "encryptionkey@0"
I20260927 00:37:58.374231 31964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:58.376511 31979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:58.377650 31964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:37:58.377769 31964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "30aaf91af22a49c88db46c21dda140fd"
format_stamp: "Formatted at 2026-09-27 00:37:58 on dist-test-slave-6gxc"
server_key: "c818c28240bc88b0923020b338f956f1"
server_key_iv: "d91e3d315edb48e3a82af0bcc92cc13b"
server_key_version: "encryptionkey@0"
I20260927 00:37:58.377862 31964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:58.391536 31964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:58.394222 31964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:58.394410 31964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:58.401037 31964 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:41531
I20260927 00:37:58.401109 32031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:41531 every 8 connection(s)
I20260927 00:37:58.401911 31964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:58.402448 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 31964
I20260927 00:37:58.402604 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:58.402842 10287 external_mini_cluster.cc:1428] Setting key e232e8a86a96a29ab81a0a9912d37cdb
I20260927 00:37:58.405018 32032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:58.410392 32032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd: Bootstrap starting.
I20260927 00:37:58.411787 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:58.404440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42923 (local address 127.10.11.254:41531)
0927 00:37:58.404833 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:58.404851 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:58.404973 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:37:58.405730 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:58.405745 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:58.406051 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:58.406433 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:58.406814 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:58.407783 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:58.408710 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:58.409163 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:58.409336 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:58.410207 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:58.410212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:58.410223 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:58.410630 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:58.410681 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:58.410703 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:58.410839 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:58.411014 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:37:58.411145 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":151,"threads_started":1}
I20260927 00:37:58.412545 32032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:58.413230 32032 log.cc:824] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:58.414625 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:58.415108 32032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd: No bootstrap required, opened a new log
I20260927 00:37:58.418684 32032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } }
I20260927 00:37:58.418905 32032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:58.418982 32032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30aaf91af22a49c88db46c21dda140fd, State: Initialized, Role: FOLLOWER
I20260927 00:37:58.419466 32032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } }
I20260927 00:37:58.419646 32032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:58.419723 32032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:58.419826 32032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:58.420322 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:58.416570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43691 (local address 127.10.11.254:41531)
0927 00:37:58.416676 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:58.416681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:58.416845 (+ 164us) server_negotiation.cc:407] Connection header received
0927 00:37:58.416966 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:58.416969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:58.417012 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:58.417096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:58.417356 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:58.417836 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:58.418694 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:37:58.418984 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:37:58.419101 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:37:58.419894 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:58.419898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:58.419901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:58.419915 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:58.419940 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:58.419953 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:58.420048 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:58.420163 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:37:58.420201 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:37:58.420796 32032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } }
I20260927 00:37:58.421120 32032 leader_election.cc:302] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30aaf91af22a49c88db46c21dda140fd; no voters:
I20260927 00:37:58.421432 32032 leader_election.cc:288] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:58.421505 32042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:58.421741 32042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [term 1 LEADER]: Becoming Leader. State: Replica: 30aaf91af22a49c88db46c21dda140fd, State: Running, Role: LEADER
I20260927 00:37:58.422061 32042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } }
I20260927 00:37:58.422598 32032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:58.423837 32043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30aaf91af22a49c88db46c21dda140fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } } }
I20260927 00:37:58.423870 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30aaf91af22a49c88db46c21dda140fd. Latest consensus state: current_term: 1 leader_uuid: "30aaf91af22a49c88db46c21dda140fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30aaf91af22a49c88db46c21dda140fd" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 41531 } } }
I20260927 00:37:58.423959 32043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:58.423965 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:58.424314 32052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:58.427409 32052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:58.432822 32052 catalog_manager.cc:1347] Generated new cluster ID: 9375ef2829774f5d859784bf6f6d80ba
I20260927 00:37:58.432896 32052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:58.445188 32052 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:58.446069 32052 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:58.458305 32052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30aaf91af22a49c88db46c21dda140fd: Generated new TSK 0
I20260927 00:37:58.458918 32052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:58.468272 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 31964
W20260927 00:37:58.473979 31949 connection.cc:535] client connection to 127.10.11.254:41531 recv error: Network error: failed to read from TLS socket (remote: 127.10.11.254:41531): Connection reset by peer (error 104)
2026-09-27T00:37:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:58 dist-test-slave-6gxc krb5kdc[32063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:58 dist-test-slave-6gxc krb5kdc[32063](info): set up 2 sockets
Sep 27 00:37:58 dist-test-slave-6gxc krb5kdc[32063](info): commencing operation
krb5kdc: starting...
W20260927 00:37:59.498265 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:37:59 dist-test-slave-6gxc krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:37:59.611515 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42849
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:37:59.715229 32076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:37:59.715451 32076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:37:59.715490 32076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:37:59.719383 32076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:37:59.719434 32076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:37:59.719453 32076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:37:59.719470 32076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:37:59.719484 32076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:37:59.724466 32076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:37:59.725301 32076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:37:59.726264 32076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:37:59.734055 32082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:59.734112 32081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:37:59.734093 32084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:37:59.734292 32076 server_base.cc:1034] running on GCE node
I20260927 00:37:59.734792 32076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:37:59.735797 32076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:37:59.736987 32076 hybrid_clock.cc:648] HybridClock initialized: now 1790469479736958 us; error 40 us; skew 500 ppm
Sep 27 00:37:59 dist-test-slave-6gxc krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:37:59.739742 32076 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:37:59.740511 32076 webserver.cc:460] Webserver started at http://127.10.11.254:34021/ using document root <none> and password file <none>
I20260927 00:37:59.741029 32076 fs_manager.cc:359] Metadata directory not provided
I20260927 00:37:59.741124 32076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:37:59.741339 32076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:37:59.743240 32076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "74557dba6fda40faab2068add5602964"
format_stamp: "Formatted at 2026-09-27 00:37:59 on dist-test-slave-6gxc"
server_key: "ac4c90814868f7be3cc5b007237678c6"
server_key_iv: "467466cde08b9f80575e05b0f843b845"
server_key_version: "encryptionkey@0"
I20260927 00:37:59.743772 32076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "74557dba6fda40faab2068add5602964"
format_stamp: "Formatted at 2026-09-27 00:37:59 on dist-test-slave-6gxc"
server_key: "ac4c90814868f7be3cc5b007237678c6"
server_key_iv: "467466cde08b9f80575e05b0f843b845"
server_key_version: "encryptionkey@0"
I20260927 00:37:59.747217 32076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:37:59.749454 32091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:37:59.750640 32076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:37:59.750773 32076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "74557dba6fda40faab2068add5602964"
format_stamp: "Formatted at 2026-09-27 00:37:59 on dist-test-slave-6gxc"
server_key: "ac4c90814868f7be3cc5b007237678c6"
server_key_iv: "467466cde08b9f80575e05b0f843b845"
server_key_version: "encryptionkey@0"
I20260927 00:37:59.750876 32076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:37:59.755086 32076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:37:59.757841 32076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:37:59.758023 32076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:37:59.764635 32076 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42849
I20260927 00:37:59.764710 32143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42849 every 8 connection(s)
I20260927 00:37:59.765475 32076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:37:59.766449 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32076
I20260927 00:37:59.766593 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:37:59.766865 10287 external_mini_cluster.cc:1428] Setting key 8666baab6242dd9416ef9a2d095c52ec
I20260927 00:37:59.768409 32144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:37:59.771548 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:59.768103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43215 (local address 127.10.11.254:42849)
0927 00:37:59.768580 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:59.768597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:59.768630 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:37:59.769205 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:59.769219 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:59.769465 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:59.769782 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:59.770482 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:59.770488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:59.770502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:59.770520 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:59.770563 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:59.770582 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:59.770723 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:59.770794 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:37:59.770931 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260927 00:37:59.774186 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:37:59.774253 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964: Bootstrap starting.
I20260927 00:37:59.776441 32144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964: Neither blocks nor log segments found. Creating new log.
I20260927 00:37:59.777310 32144 log.cc:824] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964: Log is configured to *not* fsync() on all Append() calls
I20260927 00:37:59.777593 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:37:59.776104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33483 (local address 127.10.11.254:42849)
0927 00:37:59.776219 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:37:59.776223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:37:59.776406 (+ 183us) server_negotiation.cc:407] Connection header received
0927 00:37:59.776483 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:37:59.776487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:37:59.776522 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:37:59.776606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:37:59.777223 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:37:59.777226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:37:59.777229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:37:59.777240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:37:59.777265 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:37:59.777277 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:37:59.777367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:37:59.777444 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 00:37:59.777484 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:37:59.779188 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964: No bootstrap required, opened a new log
I20260927 00:37:59.782366 32144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } }
I20260927 00:37:59.782558 32144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:37:59.782593 32144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74557dba6fda40faab2068add5602964, State: Initialized, Role: FOLLOWER
I20260927 00:37:59.782998 32144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } }
I20260927 00:37:59.783164 32144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:37:59.783236 32144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:37:59.783339 32144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:37:59.784127 32144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } }
I20260927 00:37:59.784443 32144 leader_election.cc:302] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74557dba6fda40faab2068add5602964; no voters:
I20260927 00:37:59.784729 32144 leader_election.cc:288] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:37:59.784842 32154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:37:59.785043 32154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [term 1 LEADER]: Becoming Leader. State: Replica: 74557dba6fda40faab2068add5602964, State: Running, Role: LEADER
I20260927 00:37:59.785394 32154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } }
I20260927 00:37:59.785791 32144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:37:59.786931 32155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74557dba6fda40faab2068add5602964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } } }
I20260927 00:37:59.787052 32155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:59.787396 32156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74557dba6fda40faab2068add5602964. Latest consensus state: current_term: 1 leader_uuid: "74557dba6fda40faab2068add5602964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74557dba6fda40faab2068add5602964" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42849 } } }
I20260927 00:37:59.788067 32163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:37:59.788174 32156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964 [sys.catalog]: This master's current role is: LEADER
I20260927 00:37:59.790603 32163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:37:59.795404 32163 catalog_manager.cc:1347] Generated new cluster ID: b3b1e0d1f2f14023b89ddcc61ee4e02e
I20260927 00:37:59.795473 32163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:37:59.822861 32163 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:37:59.823187 32163 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:37:59.832226 32163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74557dba6fda40faab2068add5602964: Generated new TSK 0
I20260927 00:37:59.832890 32163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:37:59.847108 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32076
2026-09-27T00:37:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:37:59 dist-test-slave-6gxc krb5kdc[32175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:37:59 dist-test-slave-6gxc krb5kdc[32175](info): set up 2 sockets
Sep 27 00:37:59 dist-test-slave-6gxc krb5kdc[32175](info): commencing operation
krb5kdc: starting...
W20260927 00:38:00.878302 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:38:00 dist-test-slave-6gxc krb5kdc[32175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:38:00.990869 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43591
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:01.098770 32188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:01.098985 32188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:01.099023 32188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:01.102928 32188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:01.102973 32188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:01.102993 32188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:01.103008 32188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:01.103067 32188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:01.107988 32188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:01.108875 32188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:01.109800 32188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:01.117661 32194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:01.117652 32193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:01.117654 32196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:01.117906 32188 server_base.cc:1034] running on GCE node
I20260927 00:38:01.118592 32188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:01.119537 32188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:01.120740 32188 hybrid_clock.cc:648] HybridClock initialized: now 1790469481120727 us; error 33 us; skew 500 ppm
Sep 27 00:38:01 dist-test-slave-6gxc krb5kdc[32175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:01.127918 32188 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:38:01.128769 32188 webserver.cc:460] Webserver started at http://127.10.11.254:38409/ using document root <none> and password file <none>
I20260927 00:38:01.129271 32188 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:01.129357 32188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:01.129577 32188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:01.131479 32188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "a5132ba1b443422982a7d28c8a421276"
format_stamp: "Formatted at 2026-09-27 00:38:01 on dist-test-slave-6gxc"
server_key: "b1e847c5d63f3bfb66b99be07aeffd97"
server_key_iv: "b39641c4c2b51e26fc348bb41cb45fb8"
server_key_version: "encryptionkey@0"
I20260927 00:38:01.132022 32188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "a5132ba1b443422982a7d28c8a421276"
format_stamp: "Formatted at 2026-09-27 00:38:01 on dist-test-slave-6gxc"
server_key: "b1e847c5d63f3bfb66b99be07aeffd97"
server_key_iv: "b39641c4c2b51e26fc348bb41cb45fb8"
server_key_version: "encryptionkey@0"
I20260927 00:38:01.135493 32188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:38:01.137672 32203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:01.138931 32188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:01.139083 32188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "a5132ba1b443422982a7d28c8a421276"
format_stamp: "Formatted at 2026-09-27 00:38:01 on dist-test-slave-6gxc"
server_key: "b1e847c5d63f3bfb66b99be07aeffd97"
server_key_iv: "b39641c4c2b51e26fc348bb41cb45fb8"
server_key_version: "encryptionkey@0"
I20260927 00:38:01.139194 32188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:01.156884 32188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:01.159914 32188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:01.160104 32188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:01.166841 32188 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:43591
I20260927 00:38:01.166854 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:43591 every 8 connection(s)
I20260927 00:38:01.167724 32188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:01.170223 32256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:01.176011 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32188
I20260927 00:38:01.176141 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:38:01.176446 10287 external_mini_cluster.cc:1428] Setting key 9bc26deffc1511d14c93b1ca50c5d7bd
I20260927 00:38:01.176952 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: Bootstrap starting.
I20260927 00:38:01.179252 32256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:01.179958 32256 log.cc:824] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:01.181936 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: No bootstrap required, opened a new log
Sep 27 00:38:01 dist-test-slave-6gxc krb5kdc[32175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:01.185715 32256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } }
I20260927 00:38:01.185953 32256 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:01.186031 32256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5132ba1b443422982a7d28c8a421276, State: Initialized, Role: FOLLOWER
I20260927 00:38:01.186494 32256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } }
I20260927 00:38:01.186673 32256 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:01.186761 32256 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:01.186859 32256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:01.187623 32256 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } }
I20260927 00:38:01.187939 32256 leader_election.cc:302] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5132ba1b443422982a7d28c8a421276; no voters:
I20260927 00:38:01.188253 32256 leader_election.cc:288] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:01.188349 32261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:01.188506 32261 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [term 1 LEADER]: Becoming Leader. State: Replica: a5132ba1b443422982a7d28c8a421276, State: Running, Role: LEADER
I20260927 00:38:01.188835 32261 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } }
I20260927 00:38:01.189419 32256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:01.190467 32262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5132ba1b443422982a7d28c8a421276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } } }
I20260927 00:38:01.190601 32262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:01.190922 32263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5132ba1b443422982a7d28c8a421276. Latest consensus state: current_term: 1 leader_uuid: "a5132ba1b443422982a7d28c8a421276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5132ba1b443422982a7d28c8a421276" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43591 } } }
I20260927 00:38:01.191087 32263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:01.191653 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:01.177652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41117 (local address 127.10.11.254:43591)
0927 00:38:01.178454 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:01.178468 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:01.178500 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:38:01.179071 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:01.179096 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:01.179464 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:01.179776 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:01.180630 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:01.182092 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:01.182955 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:01.183656 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:01.183808 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:01.186426 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:01.186431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:01.186443 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:01.186468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:01.188265 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:01.188844 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:01.188849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:01.188854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:01.188975 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:01.189267 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:01.189271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:01.189273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:01.189710 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:01.189934 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:01.190230 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 00:38:01.190533 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":124,"threads_started":1}
W20260927 00:38:01.194514 32276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:38:01.194557 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:38:01.194594 32276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:01.194665 32270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:01.195570 32270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:01.201277 32270 catalog_manager.cc:1347] Generated new cluster ID: 57c24fd10b0c41738181936bfd07bc1a
I20260927 00:38:01.201371 32270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:01.206148 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:01.196685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37343 (local address 127.10.11.254:43591)
0927 00:38:01.196785 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:01.196789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:01.197026 (+ 237us) server_negotiation.cc:407] Connection header received
0927 00:38:01.197131 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:01.197134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:01.197178 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:01.197258 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:01.198030 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:01.199130 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:01.200102 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:01.200405 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:01.200671 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:01.202859 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:01.202862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:01.202865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:01.202889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:01.204443 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:01.204913 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:01.204916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:01.204918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:01.204996 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:01.205280 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:01.205283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:01.205285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:01.205456 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:01.205576 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:01.205834 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:38:01.205922 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:38:01.207264 32270 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:01.208364 32270 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:01.221534 32270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5132ba1b443422982a7d28c8a421276: Generated new TSK 0
I20260927 00:38:01.222225 32270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:38:01.232699 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32188
2026-09-27T00:38:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:01 dist-test-slave-6gxc krb5kdc[32287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:01 dist-test-slave-6gxc krb5kdc[32287](info): set up 2 sockets
Sep 27 00:38:01 dist-test-slave-6gxc krb5kdc[32287](info): commencing operation
krb5kdc: starting...
W20260927 00:38:02.266450 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:38:02 dist-test-slave-6gxc krb5kdc[32287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:38:02.382706 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42773
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:02.487443 32300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:02.487677 32300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:02.487717 32300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:02.491671 32300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:02.491719 32300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:02.491739 32300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:02.491755 32300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:02.491807 32300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:02.496770 32300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:42773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:02.497641 32300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:02.498594 32300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:02.506456 32308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:02.506352 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:02.506348 32306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:02.506551 32300 server_base.cc:1034] running on GCE node
I20260927 00:38:02.507121 32300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:02.508003 32300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:02.509178 32300 hybrid_clock.cc:648] HybridClock initialized: now 1790469482509149 us; error 40 us; skew 500 ppm
Sep 27 00:38:02 dist-test-slave-6gxc krb5kdc[32287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:02.511878 32300 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:38:02.512717 32300 webserver.cc:460] Webserver started at http://127.10.11.254:45063/ using document root <none> and password file <none>
I20260927 00:38:02.513218 32300 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:02.513274 32300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:02.513463 32300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:02.515388 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "41e55b6b3d924fea9ce8edd5c1355634"
format_stamp: "Formatted at 2026-09-27 00:38:02 on dist-test-slave-6gxc"
server_key: "91b8afb971859446cecc7bd2c0151bb8"
server_key_iv: "2ce3f643c0bf64a66382be4db6a80a63"
server_key_version: "encryptionkey@0"
I20260927 00:38:02.515904 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "41e55b6b3d924fea9ce8edd5c1355634"
format_stamp: "Formatted at 2026-09-27 00:38:02 on dist-test-slave-6gxc"
server_key: "91b8afb971859446cecc7bd2c0151bb8"
server_key_iv: "2ce3f643c0bf64a66382be4db6a80a63"
server_key_version: "encryptionkey@0"
I20260927 00:38:02.519253 32300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:38:02.521411 32315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:02.522635 32300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:38:02.522766 32300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "41e55b6b3d924fea9ce8edd5c1355634"
format_stamp: "Formatted at 2026-09-27 00:38:02 on dist-test-slave-6gxc"
server_key: "91b8afb971859446cecc7bd2c0151bb8"
server_key_iv: "2ce3f643c0bf64a66382be4db6a80a63"
server_key_version: "encryptionkey@0"
I20260927 00:38:02.522868 32300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:02.540581 32300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:02.543087 32300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:02.543268 32300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:02.551220 32300 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:42773
I20260927 00:38:02.551290 32367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:42773 every 8 connection(s)
I20260927 00:38:02.552155 32300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:02.554684 32368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:02.557940 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32300
I20260927 00:38:02.558154 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:38:02.558439 10287 external_mini_cluster.cc:1428] Setting key bb9285935bafbe6ce4e651f8ea3f3192
I20260927 00:38:02.559994 32368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634: Bootstrap starting.
I20260927 00:38:02.562227 32368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:02.562911 32368 log.cc:824] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:02.564587 32368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634: No bootstrap required, opened a new log
I20260927 00:38:02.567086 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:02.559814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53693 (local address 127.10.11.254:42773)
0927 00:38:02.560234 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:02.560250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:02.560284 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:38:02.560873 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:02.560902 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:02.561214 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:02.561485 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:02.561853 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:02.562961 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:02.563923 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:02.564408 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:02.564586 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:02.565417 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:02.565424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:02.565435 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:38:02.565787 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:02.565839 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:38:02.565860 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:02.565977 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:02.566140 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:38:02.566421 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260927 00:38:02.568282 32368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } }
I20260927 00:38:02.568527 32368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:02.568590 32368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e55b6b3d924fea9ce8edd5c1355634, State: Initialized, Role: FOLLOWER
I20260927 00:38:02.568993 32368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } }
I20260927 00:38:02.569182 32368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:02.569254 32368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:02.569358 32368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:02.569785 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:38:02.570380 32368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } }
I20260927 00:38:02.570698 32368 leader_election.cc:302] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41e55b6b3d924fea9ce8edd5c1355634; no voters:
I20260927 00:38:02.571010 32368 leader_election.cc:288] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:02.571072 32374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:02.571295 32374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [term 1 LEADER]: Becoming Leader. State: Replica: 41e55b6b3d924fea9ce8edd5c1355634, State: Running, Role: LEADER
I20260927 00:38:02.571631 32374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } }
I20260927 00:38:02.572180 32368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:02.573194 32378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41e55b6b3d924fea9ce8edd5c1355634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } } }
I20260927 00:38:02.573331 32378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:02.573702 32387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:02.573675 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41e55b6b3d924fea9ce8edd5c1355634. Latest consensus state: current_term: 1 leader_uuid: "41e55b6b3d924fea9ce8edd5c1355634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e55b6b3d924fea9ce8edd5c1355634" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 42773 } } }
I20260927 00:38:02.573875 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:02.576723 32387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:02.577750 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:02.572102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42005 (local address 127.10.11.254:42773)
0927 00:38:02.572196 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:02.572201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:02.572217 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:38:02.572349 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:02.572351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:02.572392 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:02.572484 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:02.574560 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:02.575306 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:02.576097 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:02.576458 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:02.576556 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:02.577364 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:02.577370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:02.577375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:38:02.577388 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:02.577416 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:38:02.577430 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:02.577505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:02.577582 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 00:38:02.577626 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 00:38:02.582762 32387 catalog_manager.cc:1347] Generated new cluster ID: 1f0129251a2b463e829c4d1bb574f7eb
I20260927 00:38:02.582846 32387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:02.587908 32387 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:02.588826 32387 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:02.592989 32387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41e55b6b3d924fea9ce8edd5c1355634: Generated new TSK 0
I20260927 00:38:02.593591 32387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:38:02.600976 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32300
2026-09-27T00:38:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:02 dist-test-slave-6gxc krb5kdc[32399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:02 dist-test-slave-6gxc krb5kdc[32399](info): set up 2 sockets
Sep 27 00:38:02 dist-test-slave-6gxc krb5kdc[32399](info): commencing operation
krb5kdc: starting...
W20260927 00:38:03.634322 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:38:03 dist-test-slave-6gxc krb5kdc[32399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:38:03.748293 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43855
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:43855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:03.853264 32412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:03.853506 32412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:03.853546 32412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:03.857522 32412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:03.857570 32412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:03.857589 32412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:03.857605 32412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:03.857662 32412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:03.862641 32412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:43855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:43855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:03.863576 32412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:03.864516 32412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:03.872687 32412 server_base.cc:1034] running on GCE node
W20260927 00:38:03.872666 32417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:03.872722 32420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:03.872685 32418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:03.873572 32412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:03.874724 32412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:03.875937 32412 hybrid_clock.cc:648] HybridClock initialized: now 1790469483875878 us; error 44 us; skew 500 ppm
Sep 27 00:38:03 dist-test-slave-6gxc krb5kdc[32399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:03.878685 32412 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:38:03.879657 32412 webserver.cc:460] Webserver started at http://127.10.11.254:43867/ using document root <none> and password file <none>
I20260927 00:38:03.880209 32412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:03.880271 32412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:03.880522 32412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:03.882629 32412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "ae34e4427592415d8267ac384c1a2c0d"
format_stamp: "Formatted at 2026-09-27 00:38:03 on dist-test-slave-6gxc"
server_key: "3676d410c2e010e4db407ef2a123baf3"
server_key_iv: "30bb3cb9f1ab81ba6e71ceb862bdb7e5"
server_key_version: "encryptionkey@0"
I20260927 00:38:03.883205 32412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "ae34e4427592415d8267ac384c1a2c0d"
format_stamp: "Formatted at 2026-09-27 00:38:03 on dist-test-slave-6gxc"
server_key: "3676d410c2e010e4db407ef2a123baf3"
server_key_iv: "30bb3cb9f1ab81ba6e71ceb862bdb7e5"
server_key_version: "encryptionkey@0"
I20260927 00:38:03.886945 32412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:03.889341 32427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:03.890618 32412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:38:03.890751 32412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "ae34e4427592415d8267ac384c1a2c0d"
format_stamp: "Formatted at 2026-09-27 00:38:03 on dist-test-slave-6gxc"
server_key: "3676d410c2e010e4db407ef2a123baf3"
server_key_iv: "30bb3cb9f1ab81ba6e71ceb862bdb7e5"
server_key_version: "encryptionkey@0"
I20260927 00:38:03.890853 32412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:03.895103 32412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:03.897782 32412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:03.897956 32412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:03.904597 32412 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:43855
I20260927 00:38:03.904623 32479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:43855 every 8 connection(s)
I20260927 00:38:03.905402 32412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:03.907938 32480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:03.913162 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d: Bootstrap starting.
I20260927 00:38:03.913534 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32412
I20260927 00:38:03.913668 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:38:03.913938 10287 external_mini_cluster.cc:1428] Setting key 1c5cfe3ae8ca3acef16a54d88b0990d9
I20260927 00:38:03.915725 32480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:03.916558 32480 log.cc:824] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:03.918578 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d: No bootstrap required, opened a new log
I20260927 00:38:03.919155 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:03.915199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60541 (local address 127.10.11.254:43855)
0927 00:38:03.915598 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:03.915618 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:03.915659 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:38:03.916494 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:03.916516 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:03.916859 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:03.917224 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:03.917948 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:03.917954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:03.917968 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:38:03.917986 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:03.918033 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:38:03.918051 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:03.918200 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:03.918304 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 00:38:03.918448 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260927 00:38:03.921849 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:38:03.922624 32480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } }
I20260927 00:38:03.922890 32480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:03.922940 32480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae34e4427592415d8267ac384c1a2c0d, State: Initialized, Role: FOLLOWER
I20260927 00:38:03.923421 32480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } }
I20260927 00:38:03.923646 32486 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:03.923805 32486 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } }
I20260927 00:38:03.923604 32480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:03.924208 32480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:03.924204 32486 leader_election.cc:302] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae34e4427592415d8267ac384c1a2c0d; no voters:
I20260927 00:38:03.924325 32480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:03.924546 32486 leader_election.cc:288] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 00:38:03.925210 32480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } }
I20260927 00:38:03.925356 32480 leader_election.cc:302] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae34e4427592415d8267ac384c1a2c0d; no voters:
I20260927 00:38:03.925469 32480 leader_election.cc:288] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:03.925499 32486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:03.925661 32486 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 1 LEADER]: Becoming Leader. State: Replica: ae34e4427592415d8267ac384c1a2c0d, State: Running, Role: LEADER
I20260927 00:38:03.925804 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:03.924008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60539 (local address 127.10.11.254:43855)
0927 00:38:03.924125 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:03.924129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:03.924323 (+ 194us) server_negotiation.cc:407] Connection header received
0927 00:38:03.924506 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:03.924511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:03.924546 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:03.924618 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:03.925372 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:03.925375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:03.925377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:38:03.925388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:03.925412 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:38:03.925423 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:03.925521 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:03.925650 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:38:03.925685 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:38:03.925928 32486 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } }
I20260927 00:38:03.926610 32480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:03.927032 32491 raft_consensus.cc:2741] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [term 1 LEADER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 00:38:03.927480 32493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae34e4427592415d8267ac384c1a2c0d. Latest consensus state: current_term: 1 leader_uuid: "ae34e4427592415d8267ac384c1a2c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } } }
I20260927 00:38:03.927629 32486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae34e4427592415d8267ac384c1a2c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae34e4427592415d8267ac384c1a2c0d" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 43855 } } }
I20260927 00:38:03.927767 32486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:03.927685 32493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:03.928164 32500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:03.931133 32500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:03.936566 32500 catalog_manager.cc:1347] Generated new cluster ID: 1b902177a42d45fdad3fec0930eea105
I20260927 00:38:03.936640 32500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:03.946625 32500 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:03.946906 32500 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:03.951763 32500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae34e4427592415d8267ac384c1a2c0d: Generated new TSK 0
I20260927 00:38:03.952365 32500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:38:03.961707 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32412
2026-09-27T00:38:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1360 ms)
[----------] 18 tests from AuthTokenIssuingTest (24858 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:04 dist-test-slave-6gxc krb5kdc[32512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:04 dist-test-slave-6gxc krb5kdc[32512](info): set up 2 sockets
Sep 27 00:38:04 dist-test-slave-6gxc krb5kdc[32512](info): commencing operation
krb5kdc: starting...
W20260927 00:38:04.998287 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:38:05.108696 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36895
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:05.214785 32525 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:05.215018 32525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:05.215040 32525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:05.215076 32525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:05.218873 32525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:05.218914 32525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:05.218932 32525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:05.218948 32525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:05.218963 32525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:05.223820 32525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:05.224658 32525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:05.225512 32525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:05.233245 32530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:05.233278 32531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:05.233525 32533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:05.233507 32525 server_base.cc:1034] running on GCE node
I20260927 00:38:05.234091 32525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:05.235155 32525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:05.236333 32525 hybrid_clock.cc:648] HybridClock initialized: now 1790469485236310 us; error 43 us; skew 500 ppm
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:05.238976 32525 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:38:05.239802 32525 webserver.cc:460] Webserver started at http://127.10.11.254:45431/ using document root <none> and password file <none>
I20260927 00:38:05.240309 32525 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:05.240391 32525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:05.240614 32525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:05.242508 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "c975652c7d356023f4159a2c31c34e6e"
server_key_iv: "aa7214dac5465e8cfe20673a11e10764"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.243045 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "c975652c7d356023f4159a2c31c34e6e"
server_key_iv: "aa7214dac5465e8cfe20673a11e10764"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.246376 32525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:38:05.248533 32540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:05.249624 32525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:05.249761 32525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "c975652c7d356023f4159a2c31c34e6e"
server_key_iv: "aa7214dac5465e8cfe20673a11e10764"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.249858 32525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:05.271307 32525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:05.274402 32525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:05.274600 32525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:05.281368 32525 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:36895
I20260927 00:38:05.281661 32592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:36895 every 8 connection(s)
I20260927 00:38:05.282277 32525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:05.283612 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32525
I20260927 00:38:05.283735 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:38:05.284024 10287 external_mini_cluster.cc:1428] Setting key e35f4f06571f4a09de3fb0061be96444
I20260927 00:38:05.285346 32593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:05.289047 32593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.11.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.253@KRBTEST.COM" created.
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:05.303522 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.291248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36895 (local address 127.0.0.1:39140)
0927 00:38:05.291842 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.291874 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.292655 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.293299 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.293303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.293715 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.293931 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.293943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.295366 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.295369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.296091 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.296101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.296739 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.296743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.296863 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.297460 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.297474 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.298889 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.300713 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.300728 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.300736 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.300953 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.301273 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.301276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.301278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.301383 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.301931 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.301936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.302065 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.302289 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.302663 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":105,"spinlock_wait_cycles":13568,"thread_start_us":156,"threads_started":1}
I20260927 00:38:05.303522 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.291248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.10.11.254:36895)
0927 00:38:05.291651 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.291665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.291978 (+ 313us) server_negotiation.cc:407] Connection header received
0927 00:38:05.292947 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.292951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.293123 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.293303 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.294075 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.295225 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.296205 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.296590 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.296760 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.299030 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.299036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.299047 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.299077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.300605 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.301058 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.301064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.301065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.301172 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.301479 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.301482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.301484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.301808 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.301943 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.302364 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.302584 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260927 00:38:05.305735 32593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:05.306371 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.306158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32884)
0927 00:38:05.306251 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.306317 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
W20260927 00:38:05.306619 32541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
I20260927 00:38:05.307349 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:34465
--webserver_interface=127.10.11.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:38:05.307401 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.307198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32892)
0927 00:38:05.307301 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.307349 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:38:05.307550 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:34465: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
I20260927 00:38:05.356786 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } attempt: 1
I20260927 00:38:05.357594 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.357302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32900)
0927 00:38:05.357447 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.357511 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:38:05.358350 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.358158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32904)
0927 00:38:05.358259 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.358301 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:38:05.358553 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:34465: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
W20260927 00:38:05.412189 32600 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:05.412412 32600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:05.412433 32600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:05.412539 32600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:05.416473 32600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:05.416524 32600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:05.416544 32600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:05.416560 32600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:05.416575 32600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:05.421485 32600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.11.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:05.422427 32600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:05.423317 32600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:05.426757 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } attempt: 2
I20260927 00:38:05.427663 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.427290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32920)
0927 00:38:05.427460 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.427538 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:38:05.428416 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.428209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32930)
0927 00:38:05.428316 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.428361 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:38:05.428608 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:34465: Network error: Client connection negotiation failed: client connection to 127.10.11.253:34465: connect: Connection refused (error 111)
W20260927 00:38:05.431824 32606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:05.431932 32600 server_base.cc:1034] running on GCE node
W20260927 00:38:05.431821 32608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:05.431818 32605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:05.432632 32600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:05.433557 32600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:05.434723 32600 hybrid_clock.cc:648] HybridClock initialized: now 1790469485434694 us; error 38 us; skew 500 ppm
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:05.437366 32600 init.cc:377] Logged in from keytab as kudu/127.10.11.253@KRBTEST.COM (short username kudu)
I20260927 00:38:05.438298 32600 webserver.cc:460] Webserver started at http://127.10.11.253:36217/ using document root <none> and password file <none>
I20260927 00:38:05.438810 32600 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:05.438867 32600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:05.439090 32600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:05.440958 32600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data/instance:
uuid: "ef2cf198b2a54c8c906e80e123616490"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "2b1bc20d22ac32ee5abd284634c6c482"
server_key_iv: "1d16ca046d5a2a5b8a25e1d2c63d055d"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.441499 32600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal/instance:
uuid: "ef2cf198b2a54c8c906e80e123616490"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "2b1bc20d22ac32ee5abd284634c6c482"
server_key_iv: "1d16ca046d5a2a5b8a25e1d2c63d055d"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.445290 32600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:38:05.447571 32615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:05.448710 32600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:38:05.448845 32600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "ef2cf198b2a54c8c906e80e123616490"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "2b1bc20d22ac32ee5abd284634c6c482"
server_key_iv: "1d16ca046d5a2a5b8a25e1d2c63d055d"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.448948 32600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:05.460314 32600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:05.463299 32600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:05.463493 32600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:05.470257 32600 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.253:34465
I20260927 00:38:05.470579 32667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.253:34465 every 8 connection(s)
I20260927 00:38:05.471143 32600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
I20260927 00:38:05.473078 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32600
I20260927 00:38:05.473209 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-1/wal/instance
I20260927 00:38:05.473558 10287 external_mini_cluster.cc:1428] Setting key 0131e827088618c47097026c1eeceea8
I20260927 00:38:05.474329 32668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:05.478233 32668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.11.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.252@KRBTEST.COM" created.
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:05.492046 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.480054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.10.11.254:36895)
0927 00:38:05.480191 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.480194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.481115 (+ 921us) server_negotiation.cc:407] Connection header received
0927 00:38:05.482064 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.482068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.482163 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.482257 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.483305 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.483985 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.485121 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.485552 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.485674 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.488011 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.488017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.488023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.488059 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.489922 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.490502 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.490505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.490507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.490602 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.490928 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.490933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.490938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.491205 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.491320 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.491779 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.491913 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:38:05.492704 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.480463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36895 (local address 127.0.0.1:39144)
0927 00:38:05.480951 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.480989 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.481859 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.482412 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.482418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.482721 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.483159 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.483172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.484137 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.484140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.484995 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.485004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.485695 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.485698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.485831 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.486418 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.486466 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.487822 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.490080 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.490084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.490093 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.490397 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.490705 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.490707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.490708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.490826 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.491317 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.491321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.491522 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.491683 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.491899 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260927 00:38:05.494277 32668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
I20260927 00:38:05.500447 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:44253
--webserver_interface=127.10.11.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:38:05.502976 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.494740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32934)
0927 00:38:05.494868 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.494880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.494937 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.495444 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.495446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.495625 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.495705 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.495710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.496703 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.496706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.497255 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.497263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.497860 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.497863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.497929 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.498363 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.498374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.499606 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.501402 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.501408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.501411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.501597 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.501881 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.501884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.501886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.501964 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.502494 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.502497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.502580 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.502697 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.502810 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:38:05.503029 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.494830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.10.11.253:34465)
0927 00:38:05.495137 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.495146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.495164 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:38:05.495203 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.495207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.495338 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.495497 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.495803 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.496604 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.497367 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.497767 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.497848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.499739 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.499751 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.499768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.499798 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.501296 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.501701 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.501708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.501715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.501785 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.502056 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.502062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.502064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.502385 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.502522 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.502771 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.502885 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
I20260927 00:38:05.504582 32668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:05.505314 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.505008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60288)
0927 00:38:05.505144 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.505231 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:38:05.505508 32617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
I20260927 00:38:05.506426 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.506162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60300)
0927 00:38:05.506294 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.506352 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:38:05.506570 32668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
I20260927 00:38:05.519785 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } attempt: 3
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:38:05.527980 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.520285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.10.11.253:34465)
0927 00:38:05.520390 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.520393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.520406 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:38:05.520556 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.520558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.520593 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.520656 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.521104 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.521602 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.522626 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.522843 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.522917 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.524941 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.524945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.524947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.524978 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.526443 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.526857 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.526862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.526864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.526940 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.527212 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.527217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.527218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.527379 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.527492 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.527741 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.527842 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:38:05.527987 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.520223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32936)
0927 00:38:05.520351 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.520366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.520452 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.520699 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.520702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.520880 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.520991 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.520998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.521703 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.521706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.522520 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.522531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.522945 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.522948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.523059 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.523495 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.523509 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.524807 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.526557 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.526561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.526563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.526756 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.527033 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.527036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.527038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.527123 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.527483 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.527486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.527585 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.527692 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.527816 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:38:05.529016 32593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:05.530027 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.529796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60304)
0927 00:38:05.529908 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.529964 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:38:05.530992 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.530565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60306)
0927 00:38:05.530732 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.530939 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 00:38:05.531219 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
I20260927 00:38:05.555823 32668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 1
I20260927 00:38:05.556538 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.556272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60322)
0927 00:38:05.556407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.556466 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:38:05.557304 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.557091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60324)
0927 00:38:05.557199 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.557248 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:38:05.557495 32668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
I20260927 00:38:05.562386 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 1
I20260927 00:38:05.563104 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.562815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60328)
0927 00:38:05.562966 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.563029 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:38:05.563839 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.563623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60332)
0927 00:38:05.563735 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.563784 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:38:05.564002 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
W20260927 00:38:05.609256 32674 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:05.609503 32674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:05.609524 32674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:05.609609 32674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:05.613503 32674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:05.613559 32674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:05.613577 32674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:05.613593 32674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:05.613648 32674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:05.618721 32674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:36895,127.10.11.253:34465,127.10.11.252:44253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.11.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:05.619683 32674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:05.620677 32674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:05.625641 32668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 2
I20260927 00:38:05.626430 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.626177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60348)
0927 00:38:05.626306 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.626362 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:38:05.627090 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.626882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60362)
0927 00:38:05.626985 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.627041 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:38:05.627203 32668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
W20260927 00:38:05.629148 32680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:05.629176 32681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:05.629148 32683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:05.629470 32674 server_base.cc:1034] running on GCE node
I20260927 00:38:05.630020 32674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:05.631011 32674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:05.632192 32674 hybrid_clock.cc:648] HybridClock initialized: now 1790469485632173 us; error 28 us; skew 500 ppm
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:05.634871 32674 init.cc:377] Logged in from keytab as kudu/127.10.11.252@KRBTEST.COM (short username kudu)
I20260927 00:38:05.635756 32674 webserver.cc:460] Webserver started at http://127.10.11.252:33221/ using document root <none> and password file <none>
I20260927 00:38:05.636299 32674 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:05.636358 32674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:05.636583 32674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:05.638613 32674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data/instance:
uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "cb578efedae653387eab7fc37881124c"
server_key_iv: "e8832de134daba73155ecf47314c24ed"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.639159 32674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal/instance:
uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "cb578efedae653387eab7fc37881124c"
server_key_iv: "e8832de134daba73155ecf47314c24ed"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.639158 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 2
I20260927 00:38:05.639833 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.639583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60364)
0927 00:38:05.639708 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.639766 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:38:05.640553 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.640327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60366)
0927 00:38:05.640456 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.640508 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:38:05.640698 32593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:44253: Network error: Client connection negotiation failed: client connection to 127.10.11.252:44253: connect: Connection refused (error 111)
I20260927 00:38:05.642918 32674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:38:05.645169 32690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:05.646452 32674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:38:05.646616 32674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5"
format_stamp: "Formatted at 2026-09-27 00:38:05 on dist-test-slave-6gxc"
server_key: "cb578efedae653387eab7fc37881124c"
server_key_iv: "e8832de134daba73155ecf47314c24ed"
server_key_version: "encryptionkey@0"
I20260927 00:38:05.646720 32674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:05.655107 32674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:05.657670 32674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:05.657855 32674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:05.664502 32674 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.252:44253
I20260927 00:38:05.664772 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.252:44253 every 8 connection(s)
I20260927 00:38:05.665374 32674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
I20260927 00:38:05.665970 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 32674
I20260927 00:38:05.666095 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469213433099-10287-0/minicluster-data/master-2/wal/instance
I20260927 00:38:05.666370 10287 external_mini_cluster.cc:1428] Setting key e17da4d4f0cc7912548155e952ab3866
I20260927 00:38:05.668457 32743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:05.672722 32743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:05.681267 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.667650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.10.11.252:44253)
0927 00:38:05.668036 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.668055 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.668094 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:38:05.668933 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.668956 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.669329 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.669679 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.670375 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.671844 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.672685 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.673357 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.673513 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.676399 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.676407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.676420 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.676454 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.678386 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.678851 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.678860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.678865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.678999 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.679273 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.679277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.679279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.679772 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.679975 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.680264 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.680524 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260927 00:38:05.684299 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.675379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.10.11.254:36895)
0927 00:38:05.675532 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.675535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.675546 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:38:05.675581 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.675583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.675618 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.675711 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.676464 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.676947 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.678194 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.678410 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.678584 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.680797 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.680800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.680802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.680825 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.682472 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.682922 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.682927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.682929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.683009 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.683283 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.683288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.683290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.683519 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.683612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.684001 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.684129 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:38:05.684314 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.674569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:36895 (local address 127.0.0.1:39146)
0927 00:38:05.675199 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.675250 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.675368 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.675758 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.675763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.676162 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.676283 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.676313 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.677056 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.677062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.678038 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.678051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.678577 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.678579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.678660 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.679101 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.679114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.680655 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.682589 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.682598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.682603 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.682818 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.683102 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.683107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.683109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.683195 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.683667 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.683671 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.683791 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.683937 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.684109 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":153,"threads_started":1}
I20260927 00:38:05.685330 32743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:38:05.693146 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.683594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.10.11.254:36895)
0927 00:38:05.683915 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.683925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.683939 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:38:05.683975 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.683977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.684029 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.684114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.684828 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.685562 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.686491 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.686747 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.686853 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.689464 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.689466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.689468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.689488 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.691412 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.691895 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.691897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.691898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.691965 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.692252 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.692255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.692256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.692484 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.692698 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.692886 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.693001 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260927 00:38:05.693909 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.685803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:34465 (local address 127.0.0.1:32952)
0927 00:38:05.685931 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.685942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.686028 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.686314 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.686317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.686476 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.686554 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.686559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.687362 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.687365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.688266 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.688278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.688712 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.688716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.688815 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.689191 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.689209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.690331 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.692253 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.692256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.692258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.692522 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.692835 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.692838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.692840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.692924 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.693461 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.693464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.693557 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.693640 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.693767 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:38:05.693923 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.685938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.10.11.253:34465)
0927 00:38:05.686086 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.686092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.686132 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:38:05.686177 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.686179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.686214 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.686300 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.686666 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.687245 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.688394 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.688597 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.688673 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.690461 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.690467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.690472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.690514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.692156 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.692627 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.692630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.692632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.692716 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.693140 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.693143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.693144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.693355 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.693471 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.693722 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.693821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:38:05.694531 32743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:05.703102 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.694251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.10.11.253:34465)
0927 00:38:05.694548 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.694560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.694575 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:05.694718 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.694722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.694766 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.694859 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.695577 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.696084 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.696826 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.697079 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.697184 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.699610 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.699613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.699615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.699635 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.701454 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.701907 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.701910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.701912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.702011 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.702314 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.702318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.702320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.702485 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.702677 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.702815 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.702921 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260927 00:38:05.703116 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.694973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60378)
0927 00:38:05.695079 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.695092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.695166 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.695390 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.695392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.695559 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.695640 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.695646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.696823 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.696827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.697301 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.697308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.697728 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.697731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.697833 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.698296 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.698308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.699946 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.701757 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.701762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.701764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.701939 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.702208 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.702212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.702214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.702296 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.702643 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.702646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.702728 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.702829 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.702951 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":10}
I20260927 00:38:05.703503 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.695026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.10.11.252:44253)
0927 00:38:05.695113 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.695117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.695138 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:38:05.695254 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.695255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.695286 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.695346 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.695751 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.696708 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.697407 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.697629 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.697731 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.700090 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.700093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.700095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.700135 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.701652 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.702034 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.702039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.702041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.702135 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.702384 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.702389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.702390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.702544 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.702643 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.702870 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.703112 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":34}
I20260927 00:38:05.706053 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:38:05.706497 32743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: Bootstrap starting.
I20260927 00:38:05.707376 32567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39160
I20260927 00:38:05.708837 32743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:05.709535 32743 log.cc:824] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:05.711227 32743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: No bootstrap required, opened a new log
I20260927 00:38:05.713958 32743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.714223 32743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fb3fcd0d8c4e77b01109c50f16d9e5, State: Initialized, Role: FOLLOWER
I20260927 00:38:05.714529 32743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.715014 32751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.715142 32751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.715755 32743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:05.718330 32668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 3
W20260927 00:38:05.720121 32762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:05.720219 32762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:05.719115 32674 master_runner.cc:186] Error getting master registration for 127.10.11.254:36895: OK, instance_id { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" instance_seqno: 1790469485279579 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:38:05.722538 32674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:05.727182 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.718670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60382)
0927 00:38:05.718781 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.718794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.718868 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.719393 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.719396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.719569 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.719660 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.719666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.720623 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.720625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.721121 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.721129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.721663 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.721665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.721721 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.722203 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.722217 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.723736 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.725552 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.725555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.725575 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.725820 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.726175 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.726180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.726182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.726274 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.726678 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.726681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.726776 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.726895 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.727028 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 00:38:05.727195 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.718828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.10.11.252:44253)
0927 00:38:05.719184 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.719187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.719200 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:38:05.719231 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.719234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.719269 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.719347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.719782 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.720491 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.721225 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.721554 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.721659 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.723905 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.723911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.723917 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.723949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.725444 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.725935 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.725938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.725939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.726042 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.726365 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.726370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.726373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.726577 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.726667 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.726949 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.727052 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260927 00:38:05.730013 32668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: Bootstrap starting.
I20260927 00:38:05.732020 32668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:05.732715 32668 log.cc:824] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:05.734833 32668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: No bootstrap required, opened a new log
I20260927 00:38:05.737578 32668 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.737776 32668 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2cf198b2a54c8c906e80e123616490, State: Initialized, Role: FOLLOWER
I20260927 00:38:05.738063 32668 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.738535 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.738659 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.739393 32668 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:05.739852 32593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } attempt: 3
W20260927 00:38:05.745512 308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:05.745671 308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 00:38:05 dist-test-slave-6gxc krb5kdc[32512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469485, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:05.742560 32600 master_runner.cc:186] Error getting master registration for 127.10.11.254:36895: OK, instance_id { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" instance_seqno: 1790469485279579 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:38:05.746625 32600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:38:05.749487 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.740255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:44253 (local address 127.0.0.1:60386)
0927 00:38:05.740499 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:05.740513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:05.740592 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:05.741103 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:05.741108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:05.741320 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:05.741434 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.741441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.742470 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.742473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.743358 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:05.743369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.744026 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.744030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:05.744114 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.744627 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:05.744643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:05.746083 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:05.747973 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.747976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.747978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.748249 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.748533 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:05.748536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:05.748538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:05.748624 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:05.749006 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:05.749009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:05.749104 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 00:38:05.749201 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:38:05.749320 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 00:38:05.749541 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.740718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.10.11.252:44253)
0927 00:38:05.740850 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.740854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.740865 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:38:05.740903 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.740908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.740947 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.741030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.741583 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.742311 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.743496 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.743808 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.743903 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.746227 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.746231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.746234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.746259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.747855 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.748350 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.748354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.748356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.748437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.748714 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.748717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.748719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.748910 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.749018 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.749290 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.749396 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:38:05.752377 32593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: Bootstrap starting.
I20260927 00:38:05.754755 32593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:05.755366 32593 log.cc:824] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:05.756928 32593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: No bootstrap required, opened a new log
I20260927 00:38:05.759092 32593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.759256 32593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ef4f7ba3bb4dc19c5467ca214ac558, State: Initialized, Role: FOLLOWER
I20260927 00:38:05.759495 32593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.759878 311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.760006 311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.760552 32593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:05.760699 32567 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:38:05.760818 32567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:05.762639 32567 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
I20260927 00:38:05.764282 32642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef2cf198b2a54c8c906e80e123616490"
I20260927 00:38:05.764508 32642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:05.764667 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5"
I20260927 00:38:05.764873 32717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:05.765738 32642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ef4f7ba3bb4dc19c5467ca214ac558 in term 1.
I20260927 00:38:05.766074 32717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ef4f7ba3bb4dc19c5467ca214ac558 in term 1.
I20260927 00:38:05.766209 32541 leader_election.cc:302] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4ef4f7ba3bb4dc19c5467ca214ac558, ef2cf198b2a54c8c906e80e123616490; no voters:
I20260927 00:38:05.766453 32567 leader_election.cc:288] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [CANDIDATE]: Term 1 election: Requested vote from peers ef2cf198b2a54c8c906e80e123616490 (127.10.11.253:34465), 59fb3fcd0d8c4e77b01109c50f16d9e5 (127.10.11.252:44253)
I20260927 00:38:05.766492 311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:05.766661 311 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [term 1 LEADER]: Becoming Leader. State: Replica: d4ef4f7ba3bb4dc19c5467ca214ac558, State: Running, Role: LEADER
I20260927 00:38:05.766950 311 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } }
W20260927 00:38:05.768383 322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:05.768450 322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:05.768591 311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ef4f7ba3bb4dc19c5467ca214ac558. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.768698 311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:05.771088 330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:05.771970 330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:05.772761 32525 master_runner.cc:263] No leader master found from master d4ef4f7ba3bb4dc19c5467ca214ac558
I20260927 00:38:05.772816 32525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:38:05.776175 32546 tablet.cc:2369] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:05.777607 32717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [term 1 FOLLOWER]: Refusing update from remote peer d4ef4f7ba3bb4dc19c5467ca214ac558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:05.778333 323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:05.781819 32751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ef4f7ba3bb4dc19c5467ca214ac558. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.781925 32751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.782871 32642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [term 1 FOLLOWER]: Refusing update from remote peer d4ef4f7ba3bb4dc19c5467ca214ac558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:05.783427 323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:05.784929 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.768826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.10.11.253:34465)
0927 00:38:05.768944 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.768950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.775340 (+ 6390us) server_negotiation.cc:407] Connection header received
0927 00:38:05.775540 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.775544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.775591 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.775643 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.776664 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.777472 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.778648 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.779031 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.779125 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.780984 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.780988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.780990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.781014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.782897 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.783416 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.783420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.783422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.783524 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.783840 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.783846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.783851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.784094 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.784206 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.784609 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.784748 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:38:05.786664 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ef4f7ba3bb4dc19c5467ca214ac558. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.786773 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.788694 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.774079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.10.11.254:36895)
0927 00:38:05.774209 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.774213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.778322 (+ 4109us) server_negotiation.cc:407] Connection header received
0927 00:38:05.778475 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.778478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.778520 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.778573 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.779491 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.780288 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.781454 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.781848 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.781931 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.783631 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.783634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.783637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.783660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.785352 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.787388 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.787391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.787393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.787479 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.787768 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.787771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.787773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.787982 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.788104 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.788427 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.788542 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:38:05.792590 323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.792704 323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:05.793710 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:05.774780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.10.11.252:44253)
0927 00:38:05.774882 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.774886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.785369 (+ 10483us) server_negotiation.cc:407] Connection header received
0927 00:38:05.785563 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.785569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.785623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.785685 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.786472 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.787123 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.788181 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:05.788507 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:05.788583 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:05.790218 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:05.790222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:05.790224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:05.790248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:05.791917 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.792399 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.792403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.792404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.792482 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:05.792851 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:05.792855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:05.792857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:05.793058 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:05.793160 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:05.793470 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 00:38:05.793572 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:38:05.793828 32751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.793938 32751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.794970 332 mvcc.cc:205] Tried to move back new op lower bound from 7333763013736071168 to 7333763013704445952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763013736071168}]
I20260927 00:38:05.795112 335 mvcc.cc:205] Tried to move back new op lower bound from 7333763013736071168 to 7333763013704445952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763013736071168}]
I20260927 00:38:05.796602 330 catalog_manager.cc:1347] Generated new cluster ID: 2fc15a8dc11042cb8bc9976c35e0614f
I20260927 00:38:05.796663 330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:05.803397 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.803521 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:05.802073 311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef4f7ba3bb4dc19c5467ca214ac558" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 36895 } } peers { permanent_uuid: "ef2cf198b2a54c8c906e80e123616490" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 34465 } } peers { permanent_uuid: "59fb3fcd0d8c4e77b01109c50f16d9e5" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 44253 } } }
I20260927 00:38:05.804219 311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:05.808091 337 mvcc.cc:205] Tried to move back new op lower bound from 7333763013736071168 to 7333763013704445952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763013736071168}]
I20260927 00:38:05.841063 330 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:05.842334 330 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:05.858315 330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4ef4f7ba3bb4dc19c5467ca214ac558: Generated new TSK 0
I20260927 00:38:05.862864 330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:38:05.899343 32748 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.896541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.10.11.254:36895)
0927 00:38:05.896664 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.896669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.898332 (+ 1663us) server_negotiation.cc:407] Connection header received
0927 00:38:05.898528 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.898532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.898580 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.898631 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.899215 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39174: BlockingRecv error: recv got EOF from 127.0.0.1:39174 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:38:05.900609 32746 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.897210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.10.11.252:44253)
0927 00:38:05.897336 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.897341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.899792 (+ 2451us) server_negotiation.cc:407] Connection header received
0927 00:38:05.899949 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.899953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.899998 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.900048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.900500 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60404: BlockingRecv error: recv got EOF from 127.0.0.1:60404 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:38:05.901983 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.897745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.10.11.253:34465)
0927 00:38:05.897852 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.897856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.901084 (+ 3228us) server_negotiation.cc:407] Connection header received
0927 00:38:05.901293 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.901297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.901342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.901389 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.901875 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32974: BlockingRecv error: recv got EOF from 127.0.0.1:32974 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 00:38:05.906081 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.904667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.10.11.253:34465)
0927 00:38:05.904759 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.904763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.905228 (+ 465us) server_negotiation.cc:407] Connection header received
0927 00:38:05.905418 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.905421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.905498 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.905548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.905990 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32976: BlockingRecv error: recv got EOF from 127.0.0.1:32976 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:38:05.911494 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.910219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.10.11.253:34465)
0927 00:38:05.910311 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.910314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.910732 (+ 418us) server_negotiation.cc:407] Connection header received
0927 00:38:05.910895 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.910898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.910940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.910987 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.911403 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32980: BlockingRecv error: recv got EOF from 127.0.0.1:32980 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:38:05.917531 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.915913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.10.11.253:34465)
0927 00:38:05.916052 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.916056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.916531 (+ 475us) server_negotiation.cc:407] Connection header received
0927 00:38:05.916791 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.916794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.916842 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.916895 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.917438 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32982: BlockingRecv error: recv got EOF from 127.0.0.1:32982 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:38:05.925215 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.923732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.10.11.253:34465)
0927 00:38:05.923855 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.923859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.924323 (+ 464us) server_negotiation.cc:407] Connection header received
0927 00:38:05.924521 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.924524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.924571 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.924622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.925111 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32994: BlockingRecv error: recv got EOF from 127.0.0.1:32994 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 00:38:05.937294 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.935652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.10.11.253:34465)
0927 00:38:05.935764 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.935768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.936291 (+ 523us) server_negotiation.cc:407] Connection header received
0927 00:38:05.936524 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.936528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.936583 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.936639 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.937187 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33006: BlockingRecv error: recv got EOF from 127.0.0.1:33006 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 00:38:05.957617 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.955871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.10.11.253:34465)
0927 00:38:05.955984 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.955989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.956540 (+ 551us) server_negotiation.cc:407] Connection header received
0927 00:38:05.956770 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.956774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.956832 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.956896 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.957498 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33012: BlockingRecv error: recv got EOF from 127.0.0.1:33012 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:38:05.993357 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:05.992083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.10.11.253:34465)
0927 00:38:05.992255 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:05.992260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:05.992417 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:38:05.992620 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:05.992623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:05.992682 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:05.992775 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:05.993241 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33024: BlockingRecv error: recv got EOF from 127.0.0.1:33024 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 00:38:06.060985 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:06.059639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.10.11.253:34465)
0927 00:38:06.059803 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.059808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.059947 (+ 139us) server_negotiation.cc:407] Connection header received
0927 00:38:06.060135 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.060139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.060204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.060306 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:06.060828 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33030: BlockingRecv error: recv got EOF from 127.0.0.1:33030 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:38:06.192010 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:06.190805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.10.11.253:34465)
0927 00:38:06.190966 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.190971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.191100 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:38:06.191269 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.191272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.191329 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.191422 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:06.191893 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 00:38:06.451934 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:06.450718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.10.11.253:34465)
0927 00:38:06.450878 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.450882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.450995 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:38:06.451122 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.451125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.451175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.451252 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:06.451763 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33040: BlockingRecv error: recv got EOF from 127.0.0.1:33040 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:38:06.721501 32762 catalog_manager.cc:1259] Loaded cluster ID: 2fc15a8dc11042cb8bc9976c35e0614f
I20260927 00:38:06.721581 32762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: loading cluster ID for follower catalog manager: success
I20260927 00:38:06.723387 32762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: acquiring CA information for follower catalog manager: success
I20260927 00:38:06.724587 32762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59fb3fcd0d8c4e77b01109c50f16d9e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:38:06.746932 308 catalog_manager.cc:1259] Loaded cluster ID: 2fc15a8dc11042cb8bc9976c35e0614f
I20260927 00:38:06.747025 308 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: loading cluster ID for follower catalog manager: success
I20260927 00:38:06.748601 308 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: acquiring CA information for follower catalog manager: success
I20260927 00:38:06.749773 308 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef2cf198b2a54c8c906e80e123616490: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:38:06.969645 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:06.965955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.10.11.253:34465)
0927 00:38:06.966260 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.966264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.966280 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:38:06.966362 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.966364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.966514 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.966615 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:06.966887 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.967416 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.968306 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.968563 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.968653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:06.968921 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:06.969328 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:06.969450 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:06.969474 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 00:38:06.969518 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":75,"threads_started":1}
I20260927 00:38:06.975925 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:06.972662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.10.11.252:44253)
0927 00:38:06.972888 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.972892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.972913 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:38:06.972981 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.972983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.973076 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.973167 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:06.973345 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.973836 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.974694 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.974911 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.974997 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:06.975218 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:06.975539 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:06.975629 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:06.975691 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:38:06.975736 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260927 00:38:06.982923 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:06.978669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.10.11.254:36895)
0927 00:38:06.978947 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.978952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.979004 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:38:06.979231 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.979235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.979408 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.979519 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:06.979883 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.980453 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.981371 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.981606 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.981715 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:06.982002 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:06.982269 (+ 267us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:06.982363 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:06.982702 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 00:38:06.982758 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260927 00:38:06.983084 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:06.979022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.10.11.252:44253)
0927 00:38:06.979133 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.979140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.979464 (+ 324us) server_negotiation.cc:407] Connection header received
0927 00:38:06.979626 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.979632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.979680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.979750 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:06.980005 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.980717 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.981859 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.982203 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.982270 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:06.982648 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:06.982727 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:06.982792 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:06.982912 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 00:38:06.982967 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:38:06.983155 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:06.978682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.10.11.253:34465)
0927 00:38:06.978807 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:06.978811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:06.979030 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:38:06.979265 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:06.979267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:06.979321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:06.979398 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:06.979662 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.980457 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.981746 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:06.982072 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:06.982186 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:06.982497 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:06.982571 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:06.982651 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:06.982956 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:38:06.983038 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:38:06.989883 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32525
I20260927 00:38:06.997887 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32600
I20260927 00:38:07.003552 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 32674
2026-09-27T00:38:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3043 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:07 dist-test-slave-6gxc krb5kdc[423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:07 dist-test-slave-6gxc krb5kdc[423](info): set up 2 sockets
Sep 27 00:38:07 dist-test-slave-6gxc krb5kdc[423](info): commencing operation
krb5kdc: starting...
W20260927 00:38:08.038549 10287 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/test-user.keytab.
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.11.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
Entry for principal kudu/127.10.11.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab.
I20260927 00:38:08.151053 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39501
--webserver_interface=127.10.11.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:08.255484 436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:08.255724 436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:08.255748 436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:08.255827 436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:08.259714 436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:08.259758 436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:08.259778 436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:08.259793 436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:08.259845 436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:08.264725 436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.254.keytab
--principal=kudu/127.10.11.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.254:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.11.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:08.265671 436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:08.266583 436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:08.274408 436 server_base.cc:1034] running on GCE node
W20260927 00:38:08.274362 442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:08.274353 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:08.274362 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:08.275056 436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:08.276010 436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:08.277187 436 hybrid_clock.cc:648] HybridClock initialized: now 1790469488277176 us; error 24 us; skew 500 ppm
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:08.279781 436 init.cc:377] Logged in from keytab as kudu/127.10.11.254@KRBTEST.COM (short username kudu)
I20260927 00:38:08.280514 436 webserver.cc:460] Webserver started at http://127.10.11.254:38001/ using document root <none> and password file <none>
I20260927 00:38:08.280975 436 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:08.281030 436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:08.281186 436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:08.283027 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data/instance:
uuid: "0d32c63fd1774eb0af887442f33af066"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "5e21763d6293db98fb9326da480ba6c9"
server_key_iv: "2da8f9756b282d321b6aa80306916f63"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.283515 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance:
uuid: "0d32c63fd1774eb0af887442f33af066"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "5e21763d6293db98fb9326da480ba6c9"
server_key_iv: "2da8f9756b282d321b6aa80306916f63"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.286891 436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:38:08.289173 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:08.290398 436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:38:08.290531 436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal
uuid: "0d32c63fd1774eb0af887442f33af066"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "5e21763d6293db98fb9326da480ba6c9"
server_key_iv: "2da8f9756b282d321b6aa80306916f63"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.290624 436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:08.309376 436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:08.312387 436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:08.312582 436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:08.319072 436 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.254:39501
I20260927 00:38:08.319341 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.254:39501 every 8 connection(s)
I20260927 00:38:08.319959 436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:08.322575 504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:08.325911 504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.326217 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 436
I20260927 00:38:08.326339 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-0/wal/instance
I20260927 00:38:08.326601 10287 external_mini_cluster.cc:1428] Setting key 740b5c1748b9f1b2d1b90cf062218ce3
WARNING: no policy specified for kudu/127.10.11.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.253@KRBTEST.COM" created.
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:08.343585 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.328171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.10.11.254:39501)
0927 00:38:08.328692 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.328707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.328750 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:38:08.329737 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.329746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.329981 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.330206 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.330993 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.332488 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.333420 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.334043 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.334262 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.338620 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.338625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.338633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.338654 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.340703 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.341183 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.341192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.341195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.341339 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.341664 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.341667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.341669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.342206 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.342360 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.342700 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.342933 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":401,"spinlock_wait_cycles":5120,"thread_start_us":131,"threads_started":1}
I20260927 00:38:08.343591 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.328171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39501 (local address 127.0.0.1:33964)
0927 00:38:08.328604 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.328653 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.329430 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.330213 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.330217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.330615 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.330846 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.330858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.332619 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.332622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.333303 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.333312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.334245 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.334248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.334343 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.334933 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.334948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.338456 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.340832 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.340840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.340848 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.341065 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.341440 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.341442 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.341443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.341549 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.342322 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.342327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.342446 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.342601 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.342844 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"spinlock_wait_cycles":2432,"thread_start_us":126,"threads_started":1}
I20260927 00:38:08.345945 504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.346661 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.346421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57764)
0927 00:38:08.346525 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.346609 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:38:08.346863 452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
I20260927 00:38:08.347735 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.347489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57774)
0927 00:38:08.347648 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.347689 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 00:38:08.347882 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:42841: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
Entry for principal kudu/127.10.11.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab.
I20260927 00:38:08.353242 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:42841
--webserver_interface=127.10.11.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:38:08.397095 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } attempt: 1
I20260927 00:38:08.397867 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.397604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57790)
0927 00:38:08.397740 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.397799 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:38:08.398556 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.398357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57804)
0927 00:38:08.398456 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.398504 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:38:08.398693 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:42841: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
W20260927 00:38:08.458732 511 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:08.458998 511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:08.459064 511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:08.459137 511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:08.462939 511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:08.463009 511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:08.463047 511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:08.463083 511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:08.463115 511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:08.466861 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } attempt: 2
I20260927 00:38:08.467664 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.467347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57810)
0927 00:38:08.467491 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.467571 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 00:38:08.468041 511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.253.keytab
--principal=kudu/127.10.11.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.253:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.11.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:08.468303 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.468092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57826)
0927 00:38:08.468200 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.468246 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:38:08.468457 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.253:42841: Network error: Client connection negotiation failed: client connection to 127.10.11.253:42841: connect: Connection refused (error 111)
I20260927 00:38:08.468999 511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:08.469835 511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:08.477422 517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:08.477494 516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:08.477577 511 server_base.cc:1034] running on GCE node
W20260927 00:38:08.477458 519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:08.478643 511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:08.479694 511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:08.480859 511 hybrid_clock.cc:648] HybridClock initialized: now 1790469488480841 us; error 28 us; skew 500 ppm
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:08.483506 511 init.cc:377] Logged in from keytab as kudu/127.10.11.253@KRBTEST.COM (short username kudu)
I20260927 00:38:08.484273 511 webserver.cc:460] Webserver started at http://127.10.11.253:38567/ using document root <none> and password file <none>
I20260927 00:38:08.484789 511 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:08.484845 511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:08.484992 511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:08.486821 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data/instance:
uuid: "a46d3b6ce6b1491ab5d14b7895dedd55"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "15835847677b22e9e2120fecb0ca4ba2"
server_key_iv: "e89698846bd828ade6240e028bc66049"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.487290 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal/instance:
uuid: "a46d3b6ce6b1491ab5d14b7895dedd55"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "15835847677b22e9e2120fecb0ca4ba2"
server_key_iv: "e89698846bd828ade6240e028bc66049"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.490738 511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:38:08.492890 526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:08.494143 511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:08.494267 511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal
uuid: "a46d3b6ce6b1491ab5d14b7895dedd55"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "15835847677b22e9e2120fecb0ca4ba2"
server_key_iv: "e89698846bd828ade6240e028bc66049"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.494391 511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:08.507803 511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:08.510762 511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:08.510941 511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:08.517345 511 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.253:42841
I20260927 00:38:08.517665 578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.253:42841 every 8 connection(s)
I20260927 00:38:08.518237 511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/data/info.pb
I20260927 00:38:08.518441 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 511
I20260927 00:38:08.518579 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-1/wal/instance
I20260927 00:38:08.518927 10287 external_mini_cluster.cc:1428] Setting key 3fa9726d4d5108c3c83825c69ae06188
I20260927 00:38:08.520851 579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:08.524420 579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.11.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.11.252@KRBTEST.COM" created.
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:08.538393 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.526185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.10.11.254:39501)
0927 00:38:08.526335 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.526340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.527215 (+ 875us) server_negotiation.cc:407] Connection header received
0927 00:38:08.528185 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.528187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.528288 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.528387 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.529744 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.530557 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.531669 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.531904 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.532031 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.534905 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.534908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.534910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.534932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.536328 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.536927 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.536930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.536931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.537010 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.537327 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.537329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.537331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.537532 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.537608 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.538077 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.538210 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:38:08.539294 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.526551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39501 (local address 127.0.0.1:33976)
0927 00:38:08.527067 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.527111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.527970 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.528545 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.528552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.529048 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.529584 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.529599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.530686 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.530694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.531544 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.531554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.532074 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.532078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.532231 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.532841 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.532857 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.534734 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.536455 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.536463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.536482 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.536835 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.537094 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.537097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.537099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.537230 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.537628 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.537634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.537861 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.538030 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.538327 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":112,"threads_started":1}
I20260927 00:38:08.540575 579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
Entry for principal kudu/127.10.11.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab.
I20260927 00:38:08.544477 10287 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAm2zct/build/debug/bin/kudu
/tmp/dist-test-taskAm2zct/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:32965
--webserver_interface=127.10.11.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.11.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:38:08.550040 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.541167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.10.11.253:42841)
0927 00:38:08.541425 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.541435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.541458 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:38:08.541491 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.541495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.541634 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.541775 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.542191 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.543009 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.543830 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.544206 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.544327 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.546591 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.546599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.546607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.546628 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.548132 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.548699 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.548702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.548703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.548772 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.549077 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.549079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.549080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.549270 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.549362 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.549712 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.549893 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260927 00:38:08.550213 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.541030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57838)
0927 00:38:08.541152 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.541183 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.541280 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.541752 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.541756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.541930 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.542015 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.542020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.543133 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.543140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.543712 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.543720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.544324 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.544327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.544485 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.544922 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.544933 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.546445 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.548253 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.548259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.548263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.548587 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.548863 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.548870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.548876 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.548983 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.549374 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.549381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.549517 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.549657 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.549833 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":35}
I20260927 00:38:08.551785 579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.552510 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.552169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44880)
0927 00:38:08.552317 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.552408 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:38:08.552697 528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
I20260927 00:38:08.553607 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.553315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44882)
0927 00:38:08.553448 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.553526 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:38:08.553781 579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
I20260927 00:38:08.559602 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } attempt: 3
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:38:08.567315 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.560063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.10.11.253:42841)
0927 00:38:08.560154 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.560159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.560192 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:38:08.560324 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.560327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.560369 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.560445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.560815 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.561301 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.562039 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.562342 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.562445 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.564478 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.564482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.564484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.564503 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.565796 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.566282 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.566284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.566285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.566351 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.566658 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.566660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.566661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.566798 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.566865 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.567153 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.567230 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:38:08.567379 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.560009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57850)
0927 00:38:08.560134 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.560148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.560238 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.560460 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.560463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.560628 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.560711 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.560716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.561401 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.561406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.561925 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.561932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.562444 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.562448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.562557 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.562965 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.562976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.564358 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.565905 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.565909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.565911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.566191 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.566439 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.566443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.566445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.566561 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.566885 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.566888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.566988 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.567102 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.567226 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:38:08.568374 504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.568964 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.568772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44884)
0927 00:38:08.568870 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.568912 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:38:08.569643 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.569414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44898)
0927 00:38:08.569560 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.569598 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 00:38:08.569818 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
I20260927 00:38:08.600952 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 1
I20260927 00:38:08.602303 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.601965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44902)
0927 00:38:08.602165 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.602236 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:38:08.602933 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.602741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44910)
0927 00:38:08.602838 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.602886 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:38:08.602993 579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 1
W20260927 00:38:08.603087 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
I20260927 00:38:08.603585 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.603373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44914)
0927 00:38:08.603485 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.603529 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:38:08.604184 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.603989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44920)
0927 00:38:08.604081 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.604135 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:38:08.604296 579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
W20260927 00:38:08.658414 585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:38:08.658701 585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:08.658767 585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:08.658841 585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:08.662745 585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:08.662822 585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:08.662859 585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:08.662894 585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:08.662931 585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:08.667861 585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.11.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.11.254:39501,127.10.11.253:42841,127.10.11.252:32965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/krb5kdc/kudu_127.10.11.252.keytab
--principal=kudu/127.10.11.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.11.252:32965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.11.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:08.668780 585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:08.669684 585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:08.672441 579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 2
I20260927 00:38:08.673235 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.672928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44922)
0927 00:38:08.673065 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.673131 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:38:08.674003 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.673783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44936)
0927 00:38:08.673896 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.673945 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:38:08.674167 579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
W20260927 00:38:08.678057 594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:08.678149 591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:08.678155 592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:08.678231 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 2
I20260927 00:38:08.678480 585 server_base.cc:1034] running on GCE node
I20260927 00:38:08.678891 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.678623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44938)
0927 00:38:08.678750 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.678806 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:38:08.679051 585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:08.679522 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.679305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44952)
0927 00:38:08.679411 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.679459 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:38:08.679653 504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.11.252:32965: Network error: Client connection negotiation failed: client connection to 127.10.11.252:32965: connect: Connection refused (error 111)
I20260927 00:38:08.680300 585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:08.681470 585 hybrid_clock.cc:648] HybridClock initialized: now 1790469488681441 us; error 42 us; skew 500 ppm
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:08.684019 585 init.cc:377] Logged in from keytab as kudu/127.10.11.252@KRBTEST.COM (short username kudu)
I20260927 00:38:08.684804 585 webserver.cc:460] Webserver started at http://127.10.11.252:35153/ using document root <none> and password file <none>
I20260927 00:38:08.685314 585 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:08.685395 585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:08.685597 585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:08.687429 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data/instance:
uuid: "5e814740079a4452a897ecccdb8340ff"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "74bc9cc929a67a4cf9f5e9486e0477ab"
server_key_iv: "95b0a0d5b5ecba8ad4562d41356ae574"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.687930 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal/instance:
uuid: "5e814740079a4452a897ecccdb8340ff"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "74bc9cc929a67a4cf9f5e9486e0477ab"
server_key_iv: "95b0a0d5b5ecba8ad4562d41356ae574"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.691254 585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:38:08.693369 601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:08.694584 585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:38:08.694708 585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data,/tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal
uuid: "5e814740079a4452a897ecccdb8340ff"
format_stamp: "Formatted at 2026-09-27 00:38:08 on dist-test-slave-6gxc"
server_key: "74bc9cc929a67a4cf9f5e9486e0477ab"
server_key_iv: "95b0a0d5b5ecba8ad4562d41356ae574"
server_key_version: "encryptionkey@0"
I20260927 00:38:08.694803 585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:08.711977 585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:08.714831 585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:08.715009 585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:08.721603 585 rpc_server.cc:307] RPC server started. Bound to: 127.10.11.252:32965
I20260927 00:38:08.721882 653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.11.252:32965 every 8 connection(s)
I20260927 00:38:08.722504 585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/data/info.pb
I20260927 00:38:08.725023 654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:08.728407 654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.730237 10287 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAm2zct/build/debug/bin/kudu as pid 585
I20260927 00:38:08.730360 10287 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAm2zct/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469213433099-10287-0/minicluster-data/master-2/wal/instance
I20260927 00:38:08.730670 10287 external_mini_cluster.cc:1428] Setting key 5e96b6e3038c5066d3dfc362442e5d81
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:08.741321 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.730003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.10.11.254:39501)
0927 00:38:08.730175 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.730179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.731016 (+ 837us) server_negotiation.cc:407] Connection header received
0927 00:38:08.732148 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.732151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.732205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.732285 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.733132 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.733620 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.734693 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.734985 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.735062 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.737504 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.737508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.737511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.737537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.739129 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.739644 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.739649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.739651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.739749 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.740058 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.740062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.740064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.740267 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.740365 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.740904 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.741026 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:38:08.742166 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.730418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.254:39501 (local address 127.0.0.1:33990)
0927 00:38:08.730875 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.730928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.731954 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.732357 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.732362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.732747 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.732995 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.733009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.733719 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.733722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.734584 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.734592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.735120 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.735124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.735284 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.735986 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.736000 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.737374 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.739249 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.739256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.739264 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.739548 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.739836 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.739838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.739840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.739958 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.740365 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.740369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.740661 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.740832 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.741117 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":99,"threads_started":1}
I20260927 00:38:08.742166 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.731275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.10.11.252:32965)
0927 00:38:08.731625 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.731633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.731664 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:38:08.731953 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.731967 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.732134 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.732296 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.732980 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.733950 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.734815 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.735219 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.735313 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.737938 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.737944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.737952 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.737974 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.739611 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.740085 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.740090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.740092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.740166 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.740432 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.740435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.740436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.740686 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.740878 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.741084 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.741190 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":272,"spinlock_wait_cycles":1280,"thread_start_us":120,"threads_started":1}
I20260927 00:38:08.743371 654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.254@KRBTEST.COM
I20260927 00:38:08.751259 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.743783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.253:42841 (local address 127.0.0.1:57866)
0927 00:38:08.743888 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.743902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.743984 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.744257 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.744273 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.744432 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.744545 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.744551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.745343 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.745346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.745893 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.745902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.746360 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.746364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.746457 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.746872 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.746882 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.748144 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.749859 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.749862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.749864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.750044 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.750380 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.750383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.750385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.750469 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.750809 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.750812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.750892 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.750999 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.751112 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 00:38:08.751305 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.743897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.10.11.253:42841)
0927 00:38:08.744023 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.744030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.744045 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:08.744086 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.744088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.744133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.744222 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.744659 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.745241 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.746004 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.746260 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.746338 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.748287 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.748290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.748292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.748314 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.749756 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.750207 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.750210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.750212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.750285 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.750559 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.750563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.750564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.750719 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.750817 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.751059 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.751159 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:38:08.751860 654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:38:08.753063 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.744392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.10.11.254:39501)
0927 00:38:08.744527 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.744530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.744541 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:38:08.744640 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.744643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.744681 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.744757 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.745240 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.745745 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.746707 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.746997 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.747073 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.749604 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.749607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.749610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.749631 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.750995 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.751563 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.751566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.751568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.751655 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.751903 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.751908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.751910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.752114 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.752327 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.752822 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.752928 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.252@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.11.253@KRBTEST.COM
I20260927 00:38:08.759511 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.752270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44974)
0927 00:38:08.752347 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.752361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.752427 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.752587 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.752589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.752702 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.752761 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.752765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.753679 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.753682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.754268 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.754277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.754717 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.754720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.754791 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.755174 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.755187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.756513 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.758147 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.758149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.758151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.758381 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.758664 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.758667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.758669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.758756 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.759094 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.759099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.759193 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.759303 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.759390 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 00:38:08.759680 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.752318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.10.11.252:32965)
0927 00:38:08.752425 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.752427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.752437 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:38:08.752481 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.752482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.752512 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.752564 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.752860 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.753583 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.754381 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.754613 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.754708 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.756669 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.756673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.756675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.756701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.758006 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.758502 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.758505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.758506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.758574 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.758858 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.758861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.758863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.759002 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.759069 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.759363 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.759472 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:38:08.762625 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.754159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.10.11.253:42841)
0927 00:38:08.754265 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.754269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.754282 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:38:08.754353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.754356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.754389 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.754474 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.755021 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.755732 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.756619 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.756840 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.756940 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.759265 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.759267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.759269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.759307 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.761116 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.761494 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.761497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.761499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.761586 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.761853 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.761858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.761861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.762056 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.762313 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.762389 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.762481 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:38:08.763213 654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: Bootstrap starting.
I20260927 00:38:08.765341 579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 3
I20260927 00:38:08.765749 654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:08.765799 10287 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:38:08.766582 654 log.cc:824] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:08.767038 478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0d32c63fd1774eb0af887442f33af066"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34006
I20260927 00:38:08.768632 654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: No bootstrap required, opened a new log
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.253@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:08.772677 654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.772924 654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e814740079a4452a897ecccdb8340ff, State: Initialized, Role: FOLLOWER
I20260927 00:38:08.773241 654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.773615 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.765727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.10.11.252:32965)
0927 00:38:08.765829 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.765833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.765846 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:38:08.765932 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.765935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.765982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.766064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.766480 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.767128 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.768198 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.768497 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.768596 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.770425 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.770428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.770431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.770453 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.771954 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.772392 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.772395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.772397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.772474 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.772739 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.772743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.772745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.772956 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.773089 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.773370 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.773474 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:38:08.773615 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.765674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44986)
0927 00:38:08.765775 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.765788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.765858 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.766075 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.766078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.766262 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.766360 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.766368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.767228 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.767234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.768042 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.768053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.768600 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.768633 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.768724 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.769154 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.769167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.770309 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.772058 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.772062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.772063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.772295 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.772566 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.772570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.772571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.772654 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.773109 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.773112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.773201 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.773306 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.773434 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:38:08.774125 660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.774262 660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.774585 654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:08.776535 579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: Bootstrap starting.
I20260927 00:38:08.778697 579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:08.778786 504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } attempt: 3
W20260927 00:38:08.778926 671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:08.779009 671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:08.779524 579 log.cc:824] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:08.782199 579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: No bootstrap required, opened a new log
I20260927 00:38:08.781514 585 master_runner.cc:186] Error getting master registration for 127.10.11.254:39501: OK, instance_id { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" instance_seqno: 1790469488317281 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:38:08.784907 585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:38:08.786010 579 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.786278 579 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46d3b6ce6b1491ab5d14b7895dedd55, State: Initialized, Role: FOLLOWER
I20260927 00:38:08.786597 579 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.787066 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.787209 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.787858 579 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:38:08.792543 685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:08.792618 685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 00:38:08 dist-test-slave-6gxc krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469488, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.11.254@KRBTEST.COM for kudu/127.10.11.252@KRBTEST.COM
I20260927 00:38:08.794960 511 master_runner.cc:186] Error getting master registration for 127.10.11.254:39501: OK, instance_id { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" instance_seqno: 1790469488317281 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:38:08.798645 511 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:38:08.801996 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.779193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.10.11.252:32965)
0927 00:38:08.780226 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.780233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.780249 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:38:08.780280 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.780283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.780320 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.790398 (+ 10078us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.790481 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.791280 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.792388 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.792695 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.793128 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.798755 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.798758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.798761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.798784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.800356 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.800830 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.800833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.800835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.800911 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.801192 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.801195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.801196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.801382 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.801471 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.801757 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.801856 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866}
I20260927 00:38:08.801998 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.779166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.11.252:32965 (local address 127.0.0.1:44998)
0927 00:38:08.779319 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:08.779330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:08.779390 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:08.789940 (+ 10550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:08.789943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:08.790176 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:08.790286 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.790293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.791411 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.791415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.792246 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:08.792256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.792819 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.792823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:08.792908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.793408 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:08.793424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:08.798410 (+ 4986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:08.800458 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.800461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.800464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.800732 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.800997 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:08.801000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:08.801003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:08.801098 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:08.801477 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:08.801480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:08.801580 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:38:08.801690 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:38:08.801830 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 00:38:08.804982 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: Bootstrap starting.
I20260927 00:38:08.807591 504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:08.808446 504 log.cc:824] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:08.810559 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: No bootstrap required, opened a new log
I20260927 00:38:08.813319 504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.813570 504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d32c63fd1774eb0af887442f33af066, State: Initialized, Role: FOLLOWER
I20260927 00:38:08.813987 504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.814571 688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.814785 688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.815302 504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:08.815402 478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:38:08.815500 478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:08.816776 478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.817710 478 leader_election.cc:288] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [CANDIDATE]: Term 1 election: Requested vote from peers a46d3b6ce6b1491ab5d14b7895dedd55 (127.10.11.253:42841), 5e814740079a4452a897ecccdb8340ff (127.10.11.252:32965)
I20260927 00:38:08.818490 628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d32c63fd1774eb0af887442f33af066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e814740079a4452a897ecccdb8340ff"
I20260927 00:38:08.818722 628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:08.819747 553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d32c63fd1774eb0af887442f33af066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55"
I20260927 00:38:08.819940 553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:08.819998 628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d32c63fd1774eb0af887442f33af066 in term 1.
I20260927 00:38:08.820469 453 leader_election.cc:302] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d32c63fd1774eb0af887442f33af066, 5e814740079a4452a897ecccdb8340ff; no voters:
I20260927 00:38:08.821169 553 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d32c63fd1774eb0af887442f33af066 in term 1.
I20260927 00:38:08.821375 688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:08.822089 688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [term 1 LEADER]: Becoming Leader. State: Replica: 0d32c63fd1774eb0af887442f33af066, State: Running, Role: LEADER
I20260927 00:38:08.822450 688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } }
I20260927 00:38:08.824944 688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d32c63fd1774eb0af887442f33af066. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.825100 688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: This master's current role is: LEADER
W20260927 00:38:08.829046 703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:08.829130 703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:08.829208 709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:08.830068 709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:08.836086 628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [term 1 FOLLOWER]: Refusing update from remote peer 0d32c63fd1774eb0af887442f33af066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:08.836768 704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:08.839444 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.823360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.10.11.252:32965)
0927 00:38:08.823472 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.823477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.830367 (+ 6890us) server_negotiation.cc:407] Connection header received
0927 00:38:08.830510 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.830513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.830562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.830649 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.831740 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.832512 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.833685 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.834008 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.834080 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.835802 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.835806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.835809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.835834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.837572 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.838138 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.838141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.838143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.838228 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.838549 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.838552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.838554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.838752 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.838846 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.839176 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.839299 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:38:08.840883 660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d32c63fd1774eb0af887442f33af066. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.841007 660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.842478 553 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [term 1 FOLLOWER]: Refusing update from remote peer 0d32c63fd1774eb0af887442f33af066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:08.843379 704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:08.843703 688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.843813 688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:08.844417 711 mvcc.cc:205] Tried to move back new op lower bound from 7333763026265628672 to 7333763026220285952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763026265628672}]
I20260927 00:38:08.844691 660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.844805 660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.847558 712 mvcc.cc:205] Tried to move back new op lower bound from 7333763026265628672 to 7333763026220285952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763026265628672}]
I20260927 00:38:08.847906 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d32c63fd1774eb0af887442f33af066. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.848023 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.848034 709 catalog_manager.cc:1347] Generated new cluster ID: fa3f6acbd2bc42a8b6f346431a171b93
I20260927 00:38:08.848093 709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:08.848134 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.848230 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:38:08.848614 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.822494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.10.11.254:39501)
0927 00:38:08.822593 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.822597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.830369 (+ 7772us) server_negotiation.cc:407] Connection header received
0927 00:38:08.830522 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.830526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.830572 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.830622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.838418 (+ 7796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.839143 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.840204 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.840519 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.840599 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.842593 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.842599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.842602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.842628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.845631 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.846135 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.846141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.846143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.846231 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.846541 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.846544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.846546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.846763 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.846890 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.848377 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.848494 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:38:08.854562 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:08.823915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.10.11.253:42841)
0927 00:38:08.824019 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.824023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.830690 (+ 6667us) server_negotiation.cc:407] Connection header received
0927 00:38:08.830852 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.830858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.830907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.830958 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.842964 (+ 12006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.843654 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.849061 (+ 5407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:08.849360 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:08.849438 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:08.851135 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:08.851139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:08.851142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:08.851168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:08.852720 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.853285 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.853289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.853291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.853371 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:08.853662 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:08.853665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:08.853667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:08.853857 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:08.853958 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:08.854297 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 00:38:08.854404 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:38:08.862605 716 mvcc.cc:205] Tried to move back new op lower bound from 7333763026265628672 to 7333763026220285952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763026265628672}]
I20260927 00:38:08.870498 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d32c63fd1774eb0af887442f33af066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d32c63fd1774eb0af887442f33af066" member_type: VOTER last_known_addr { host: "127.10.11.254" port: 39501 } } peers { permanent_uuid: "a46d3b6ce6b1491ab5d14b7895dedd55" member_type: VOTER last_known_addr { host: "127.10.11.253" port: 42841 } } peers { permanent_uuid: "5e814740079a4452a897ecccdb8340ff" member_type: VOTER last_known_addr { host: "127.10.11.252" port: 32965 } } }
I20260927 00:38:08.870667 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:08.876552 709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:08.877691 709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:08.889037 709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d32c63fd1774eb0af887442f33af066: Generated new TSK 0
I20260927 00:38:08.889860 709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:38:08.912834 508 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.910370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.10.11.254:39501)
0927 00:38:08.910490 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.910495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.911767 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 00:38:08.911976 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.911980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.912030 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.912085 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.912707 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34022: BlockingRecv error: recv got EOF from 127.0.0.1:34022 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:38:08.914275 658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.911158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.10.11.252:32965)
0927 00:38:08.911269 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.911273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.913319 (+ 2046us) server_negotiation.cc:407] Connection header received
0927 00:38:08.913497 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.913500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.913547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.913601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.914144 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45030: BlockingRecv error: recv got EOF from 127.0.0.1:45030 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 00:38:08.916301 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.914980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.10.11.253:42841)
0927 00:38:08.915097 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.915102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.915405 (+ 303us) server_negotiation.cc:407] Connection header received
0927 00:38:08.915584 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.915587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.915635 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.915688 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.916181 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57892: BlockingRecv error: recv got EOF from 127.0.0.1:57892 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:38:08.923730 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.922451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.10.11.253:42841)
0927 00:38:08.922573 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.922578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.922783 (+ 205us) server_negotiation.cc:407] Connection header received
0927 00:38:08.923015 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.923018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.923090 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.923145 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.923634 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57902: BlockingRecv error: recv got EOF from 127.0.0.1:57902 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:38:08.930830 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.929307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.10.11.253:42841)
0927 00:38:08.929408 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.929412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.929872 (+ 460us) server_negotiation.cc:407] Connection header received
0927 00:38:08.930074 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.930077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.930210 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.930263 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.930738 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57914: BlockingRecv error: recv got EOF from 127.0.0.1:57914 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 00:38:08.936359 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.934928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.10.11.253:42841)
0927 00:38:08.935024 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.935028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.935484 (+ 456us) server_negotiation.cc:407] Connection header received
0927 00:38:08.935664 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.935668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.935714 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.935767 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.936268 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57916: BlockingRecv error: recv got EOF from 127.0.0.1:57916 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 00:38:08.943879 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.942462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.10.11.253:42841)
0927 00:38:08.942583 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.942587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.943054 (+ 467us) server_negotiation.cc:407] Connection header received
0927 00:38:08.943234 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.943237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.943283 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.943334 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.943791 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57918: BlockingRecv error: recv got EOF from 127.0.0.1:57918 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:38:08.955436 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.953975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.10.11.253:42841)
0927 00:38:08.954073 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.954078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.954561 (+ 483us) server_negotiation.cc:407] Connection header received
0927 00:38:08.954744 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.954748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.954794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.954847 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.955343 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57932: BlockingRecv error: recv got EOF from 127.0.0.1:57932 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:38:08.975661 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:08.973935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.10.11.253:42841)
0927 00:38:08.974061 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:08.974066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:08.974631 (+ 565us) server_negotiation.cc:407] Connection header received
0927 00:38:08.974852 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:08.974855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:08.974926 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:08.974985 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:08.975547 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57938: BlockingRecv error: recv got EOF from 127.0.0.1:57938 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:38:09.012404 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:09.010507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.10.11.253:42841)
0927 00:38:09.010648 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.010654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.011284 (+ 630us) server_negotiation.cc:407] Connection header received
0927 00:38:09.011520 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.011524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.011585 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.011646 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:09.012274 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57950: BlockingRecv error: recv got EOF from 127.0.0.1:57950 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 00:38:09.080157 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:09.078972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.10.11.253:42841)
0927 00:38:09.079132 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.079138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.079268 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:38:09.079432 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.079436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.079519 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.079610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:09.080041 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57960: BlockingRecv error: recv got EOF from 127.0.0.1:57960 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:38:09.211237 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:09.210272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.10.11.253:42841)
0927 00:38:09.210404 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.210407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.210555 (+ 148us) server_negotiation.cc:407] Connection header received
0927 00:38:09.210679 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.210681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.210728 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.210804 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:09.211145 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57968: BlockingRecv error: recv got EOF from 127.0.0.1:57968 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:38:09.470337 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:38:09.469325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.10.11.253:42841)
0927 00:38:09.469456 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.469460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.469491 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:38:09.469677 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.469680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.469726 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.469804 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:09.470212 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57980: BlockingRecv error: recv got EOF from 127.0.0.1:57980 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:38:09.780170 671 catalog_manager.cc:1259] Loaded cluster ID: fa3f6acbd2bc42a8b6f346431a171b93
I20260927 00:38:09.780251 671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: loading cluster ID for follower catalog manager: success
I20260927 00:38:09.782020 671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: acquiring CA information for follower catalog manager: success
I20260927 00:38:09.783164 671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e814740079a4452a897ecccdb8340ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:38:09.793757 685 catalog_manager.cc:1259] Loaded cluster ID: fa3f6acbd2bc42a8b6f346431a171b93
I20260927 00:38:09.793836 685 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: loading cluster ID for follower catalog manager: success
I20260927 00:38:09.795408 685 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: acquiring CA information for follower catalog manager: success
I20260927 00:38:09.796418 685 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a46d3b6ce6b1491ab5d14b7895dedd55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:38:09.988309 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:09.984346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.10.11.253:42841)
0927 00:38:09.984578 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.984584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.984666 (+ 82us) server_negotiation.cc:407] Connection header received
0927 00:38:09.984849 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.984852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.984972 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.985090 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:09.985361 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.985890 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:09.986777 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.987005 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:09.987114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:09.987425 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:09.987852 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:09.987987 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:09.988024 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 00:38:09.988082 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260927 00:38:09.994508 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:09.991223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.10.11.252:32965)
0927 00:38:09.991443 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.991446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.991458 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:38:09.991528 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.991530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.991656 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.991759 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:09.991926 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.992414 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:09.993147 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.993414 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:09.993490 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:09.993724 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:09.994164 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:09.994252 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:09.994346 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:38:09.994399 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260927 00:38:10.001461 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:09.997363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.10.11.253:42841)
0927 00:38:09.997479 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.997483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.997702 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:38:09.997835 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.997838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.997890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.997953 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:09.998232 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.999014 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000256 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:10.000514 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000568 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:10.001008 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:10.001077 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:10.001143 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:10.001270 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 00:38:10.001316 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:38:10.001513 797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:09.997102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.10.11.254:39501)
0927 00:38:09.997346 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.997350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.997504 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:38:09.997720 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.997724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.997923 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.998026 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:09.998407 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.999135 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000015 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:10.000240 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:10.000721 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:10.001012 (+ 291us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:10.001091 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:10.001188 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 00:38:10.001240 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260927 00:38:10.001618 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:09.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.10.11.252:32965)
0927 00:38:09.997296 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:09.997300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:09.997862 (+ 562us) server_negotiation.cc:407] Connection header received
0927 00:38:09.998021 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:09.998024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:09.998114 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:09.998167 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:09.998544 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:09.999224 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000323 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:10.000623 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:10.000714 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:10.000945 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:10.001011 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:10.001095 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:10.001450 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:38:10.001496 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:38:10.004346 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 436
I20260927 00:38:10.010987 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 511
I20260927 00:38:10.016453 10287 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAm2zct/build/debug/bin/kudu with pid 585
2026-09-27T00:38:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3012 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6056 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276578 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST